Scaling UnoPim for 10 Million Products

UnoPim is an open-source product information management (PIM) system built on Laravel that is designed to help businesses manage and organize product data efficiently.

It supports seamless distribution across sales channels and scales well with large catalogs.

UnoPim’s scalable architecture and performance-focused design enable it to handle large, complex product data with ease.

Manage attributes, descriptions, images, and technical specs across multiple channels and languages, keeping everything consistent from a single source.

Managing 10 million products requires more than just a capable app, it demands smart planning at every level.

This blog explores scaling UnoPim by speeding imports, optimizing database, using queue workers, and enhancing search with Elasticsearch to keep your PIM fast, stable, and scalable.

With around 1 million products in the system, UnoPim runs smoothly, handling database queries and search operations efficiently without relying on advanced optimizations or caching.

This demonstrates the strength and scalability of its architecture, delivering consistent performance even with large datasets.

Challenges of Scaling UnoPim

As your product catalog grows, scaling UnoPim to support millions of items isn’t just about storing more data—it’s about maintaining smooth, reliable performance.

Here are some common challenges that come with managing a large dataset:

1. Database Performance: With more products, database queries can slow down. If the system isn’t optimized, even basic operations like listing or updating products can become sluggish.
2. Data Retrieval: Pulling large sets of product data, especially across different relations or locales, can take time if indexing and optimization aren’t in place.
3. Search Optimization: A growing catalog makes fast, accurate search even more important. Without enhancements like Elasticsearch, users may face delays when trying to find what they need.
4. Server Resources: Managing millions of records puts pressure on your infrastructure. You’ll need enough CPU, memory, and storage to handle spikes in traffic or batch updates.

As product data volume increases, even small inefficiencies can multiply into major slowdowns.

Strategies for Scaling UnoPim

Database Optimization

We started by designing for scale at the data layer:

• Table Partitioning: Large tables (like product, attributes, variants) were partitioned by product type or tenant, reducing index bloat.
• Indexing Strategy: We added compound indexes for frequently queried fields and used covering indexes to minimize lookups.

Implement Efficient Data Retrieval

• Use lazy loading to load products on demand.
• Use data pagination to limit the amount of data retrieved.

Caching for Speed

To support high-performance APIs and user interfaces:

• Query Result Caching: We implemented smart caching for repetitive queries with high read volumes.
• ElasticSearch powers full-text search, advanced filtering, and faceted navigation, even with millions of SKUs.
  

Async and Queue-Driven Architecture

We avoided real-time bottlenecks by offloading heavy tasks:

• Bulk exports/imports: Data syncs run asynchronously via queues.
• Cron setup: You want to schedule a specific queued job using its job ID
• Event-driven architecture: Allows loosely coupled services to react to product updates.

Server Resource Optimization

• Increase server resources, such as CPU, memory, and storage, as needed.
• Use containerization, such as Docker, to improve resource utilization.

Indexing Strategies in UnoPim for Scalable Database Performance Using Elasticsearch

As UnoPim scaled to handle catalogs with over 10 million products, ensuring fast search, filtering, and real-time data retrieval became critical.

Traditional Relational databases often face challenges when handling large-scale data, even with indexing.

To address this, we use Elasticsearch, a distributed search engine that improves search efficiency and scalability.

In this article, we’ll break down how UnoPim uses Elasticsearch in combination with smart relational indexing to deliver scalable, high-performance product data management.

Why Elasticsearch?

Elasticsearch is a high-performance, distributed search engine designed for speed, scalability, and advanced full-text search.

When integrated with UnoPim, it enhances the platform with the following key benefits:

• Lightning-Fast Search Results: Elasticsearch enables near-instantaneous search across large catalogs, ensuring a smooth user experience even with thousands of products.
• Advanced Full-Text Search: Supports features like stemming, typo tolerance, and relevance scoring, delivering more accurate and intuitive search results.
• Real-Time Indexing: Product updates, categories, and new listings can be reflected in search results with minimal delay.

Elasticsearch solves these problems by indexing data optimally for fast search and filtering, making it ideal for PIM systems like UnoPim.

To configure Elasticsearch, please refer to the Configuration Setup documentation.

Final Performance Benchmark:

Product List: After implementing Elasticsearch, we observed significant improvements in search performance and page load time, even with a high data volume.

In the image below, the page load time is recorded at approximately 1.35 seconds, handling a dataset of 10 million products efficiently.

This benchmark confirms the effectiveness of Elasticsearch in scaling search capabilities while maintaining a fast and responsive user experience.

Category List: After implementing Elasticsearch, we observed a significant improvement in both search performance and page load time, even with a high volume of categories.

In the image below, the page load time is recorded at approximately 296 milliseconds, efficiently handling a dataset of 10K categories.

This benchmark confirms the effectiveness of Elasticsearch in scaling search capabilities while maintaining a fast and responsive user experience.

Attribute List: In the image below, the page load time is recorded at approximately 271 milliseconds, efficiently handling a dataset of 5K Attributes(without Elasticsearch).

Conclusion: Future Scalability Beyond 10 Million Products

Scaling UnoPim to handle 10 million products is both a challenging and achievable goal. With the right optimizations in place, such as:

You can ensure your UnoPim–powered store remains high-performing, even as your product catalog grows exponentially.

This guide serves as a foundation for future-proofing your UnoPim store, making it capable of scaling seamlessly as your business grows.

We hope you found this guide insightful and helpful.

Should you encounter any challenges or have further questions, feel free to leave a comment below—we’re here to assist you in optimizing and scaling your UnoPim experience to new heights!

Need help scaling your PIM? Contact our team