As I prepared to interview Nicholas Knize Ph.D., CTO of Lucenia, I reflected on my journey over the past two months since joining the company. Initially, I felt daunted by the complex realm of search analytics, despite my confidence in my job responsibilities. Lucenia, a unique startup with the slogan “[SEARCH]… on your terms,” often leaves those outside the tech industry puzzled, questioning even the basics of cloud technology. Yet, through this immersive experience, I have distilled these concepts into more accessible insights for a broader audience. My prior understanding of the cloud was limited to the necessity of expanding my phone’s storage due to an overload of photos. Now, I am beginning to grasp why Lucenia is making waves in the industry. My curiosity led me deeper into the origins of this groundbreaking venture and to discover what inspired Dr. Knize to embark on this journey.

—

Company Inception

What inspired you to start Lucenia? Was there a specific problem or gap in the market that you identified?

What inspired me to start Lucenia was my experience working with companies that prioritize complex legacy architectures over simple customer needs, which have repercussions on the broader economy. Many companies push bloated cloud solutions, leading to vendor lock-in and unnecessary features. Take search, for example. Often, customers just need the ability to explore their data securely. However, current offerings like Elastic and OpenSearch force-include features for log analytics, security analytics, or the latest buzz – vectors and Generative AI – whether needed or not. This is due to these projects’ legacy implementations and the vendors’ inability to eliminate tech debt, driving up costs and reducing performance as customers are forced to adapt to more expensive infrastructure for unused features. With Lucenia, our goal is to offer a fresh perspective on Search, prioritizing flexibility and race-car-like performance with the cost-efficiency of a sedan through minimalism. We empower customers to use only the capabilities and resources they need without being constrained by rising expenses caused by over-provisioned infrastructure. This approach boosts search performance while saving customers money and reducing friction for adding new, exciting features. In other words, “[SEARCH]…on your terms.”

‍

What is the vision and mission of Lucenia? How do you see the company making a difference in the search industry?

The vision of Lucenia is simple, we aim to deliver a suite of search and analysis tools, blending the power of open-source technologies with tailored, severless autoscaling capabilities and flexible enterprise-grade security. The current landscape of search technologies is forcing organizations to spend immense amounts of money on cloud infrastructure they don’t need. I believe Lucenia is set to revolutionize the search industry by helping organizations significantly cut these cloud expenses, making cloud search capabilities more accessible and affordable for everyone.

What personal experiences or interests drove you towards founding a search company like Lucenia?

My journey towards founding Lucenia is deeply rooted in my decades-long experience working in the Department of Defense and Intelligence Communities, particularly with forward deployed units. In these network-starved and data-hungry environments, getting critical information into the hands of forward-deployed warfighters using limited resources is essential and life-saving. I dedicated a significant part of my career to building platforms that deliver necessary information in these challenging settings. The lessons learned from these demanding technological environments have been instrumental in shaping Lucenia. I’ve applied these insights to design robust, efficient search solutions not just for the public sector but to bring them to the private sector as well, ensuring that an organization’s critical data is always accessible, reliable, and protected, no matter the context.

Technology and Innovation

What sets Lucenia apart from other search engines and technologies in the market? Can you describe some unique features or innovations?

The ability to scale down and the ability to use the right tool for the job. The architecture is designed as a microlith: short-lived functions combined with some traditional distributed design techniques to enable the platform to adjust to the demand of each user’s unique use case.

How does Lucenia leverage AI and machine learning to enhance search capabilities?

We provide a platform for AI and for AI use cases to search high volumes of high dimensional vectors. We are an enabler for AI solutions and for those that want to build next generation AI capabilities in an infrastructure and cost efficient environment.  

Are there any groundbreaking algorithms or methodologies involved?‍

Have a look at the blog post: Reducing Cloud Storage for Generative AI: Lucenia’s Approach to Vector Search

In today’s world, there’s a belief that more dimensions equals better results, particularly with generative AI trends. However, I challenge this perspective. I question the value of paying exorbitant prices for marginal increases in quality, especially if it’s not needed or “mission critical.” Consider the analogy of compressing photos: why invest significantly for a mere 5% improvement when the difference is imperceptible to the human eye? At Lucenia, we prioritize efficiency and practicality over unnecessary complexity, ensuring that our algorithms and methodologies deliver tangible value without excessive costs.

How do you balance speed, accuracy, and relevance in search results?

Our focus is on optimizing speed and performance. Our microlithic architecture is designed to scale out efficiently based on computational demand, ensuring rapid response times. When it comes to relevance, we leverage Lucene’s flexibility and offer extension points for users to incorporate their own scoring algorithms. While relevance computations can be complex and potentially slow, our microlithic design allows us to strike a balance. By breaking down tasks into small, scalable units, we can adjust resources to accommodate the complexity of relevance scoring, maintaining a high level of accuracy while delivering results with remarkable speed.

Challenges and Solutions

What were some of the biggest challenges you have faced while starting Lucenia, and how have you overcome them?

The number one challenge that we have faced has been deploying and scaling the Lucene OpenSource software in a cloud native environment whether that is self-managed hardware or a vendor provided platform. Lucene is not focused on a cloud native distributed environment, it’s a stand-alone search API developed by many different organizations building their own distributed search platforms in their own way. Lucenia’s objective is to approach this with a fresh look at cloud native serverless search, powered by Lucene data structures and concepts: one of the longest running Apache Open Source projects. What is reflected in Lucene’s implementation is the lack of supporting new cloud native architectures and software infrastructures. Things like kubernetes, Docker, cloud virtualization did not exist until at least 10 years after Lucene was created: these have their orchestration challenges in a cloud native environment, it becomes clear that the Lucene search API is not designed or implemented in that environment and we have had to work this out in our architecture design.

How does Lucenia address data security issues such as data privacy and user trust?

Lucenia provides trust through security out-of-the-box, by default; from index level to field level security. Organizations have the ability to set their own policies and ensure data is only available to those who really need access. This approach is born out of my experience in multi-level classified and “need to know” environments within the government and Intelligence Communities. Operations Security, or OPSEC, is another Lucenia feature born out of these experiences. OPSEC is the principle that individual pieces of data may seem harmless on their own, but when combined create a highly sensitive piece of aggregate information and intelligence. This is crucial for an organization to protect and even harder to detect. Take someone’s credit card as a simple example of this idea. The sixteen digit number by itself may be insignificant, but when combined with a name, expiration date, and security code, becomes highly valuable and extremely costly if compromised. And this breach of data and loss of trust is becoming increasingly common. With a 72% rise in security incidents in cloud infrastructure, organizations are looking for reliable solutions and alternatives. Lucenia is proving to be a reliable partner in this area enabling organizations to rapidly, “[SEARCH]…on your terms.”  

What challenges have you encountered in scaling the technology and infrastructure, and what strategies have you employed to solve this?

It is easy to scale up – and attractive to do so for cloud vendors reaping the financial gains from over-provisioned infrastructure. The unfortunate victim in that scenario is the customer. The end user or organization who has to  pay more because scaling down isn’t an option. Think of the iCloud user who runs out of space and has to pay more to just keep their family photos, and cloud vendors love it! They make money off users’ misfortune instead of helping the customer with tools and features that free up space to save money. Lucenia is tackling this challenge head on by scaling down, to enable customers to trim their cost and scale their storage and compute infrastructure as far down as they want to go.  

Impact and Future

How will you measure the impact and success of Lucenia in the current market?‍

At Lucenia, we believe that true success is reflected in the value we provide to our customers. Instead of focusing solely on the bottom line, we measure our success by how effectively we can reduce waste for our customers. Currently, about 30% of infrastructure is wasted, and our goal is to help organizations significantly cut those numbers. Success for Lucenia means enabling our customers to identify and reduce inefficiencies, thereby optimizing their resources. By joining us on this journey, our customers will experience substantial reductions in waste, and we will support them every step of the way in measuring and continuing to minimize these inefficiencies. To me, this approach is a win-win situation: as our customers thrive and grow by reducing their waste, so does Lucenia grow alongside them, fostering mutual growth and success.

How do you see the search industry evolving over the next 5-10 years, and what role do you envision Lucenia playing in that future?‍

Over the next 5-10 years, I see search capabilities becoming far more accessible and easier to deploy, tailor, use, and maintain, without requiring advanced expertise. Current search solutions often demand a high level of industry knowledge, whether cloud-managed or not. This complexity leads to inefficiencies in both storage and compute, and the need for specialized personnel to maintain the deployment adds another layer of cost for organizations. At Lucenia, we believe that simple tasks should remain simple, even when dealing with fluctuating compute demands. Our goal is to provide reliable and secure search capabilities that anyone can deploy and integrate seamlessly. We are leading this transformation by focusing on key differentiators such as security by default, memory-safe languages, and an autoscaling microlithic design. Lucenia aims to make advanced search technology universally accessible, efficient, and cost-effective.

Personal Insights

How would you describe your leadership style, and how has it influenced the culture at Lucenia?‍

My leadership style is fundamentally rooted in trust. Building trust within the team is crucial for fostering confidence and encouraging continuous growth. At Lucenia, this trust-based leadership has cultivated a culture where diversity of thought drives innovation. We emphasize hard work, understanding, and a results-driven attitude, all while learning from each other. This collaborative environment ensures that every team member’s unique perspective contributes to our collective success and drives us towards achieving our goals.

What qualities do you look for when building your team at Lucenia? How do you foster innovation and creativity within the company?‍

When building the team at Lucenia, I seek individuals who are creative, results-driven, consider themselves life-long learners, industry experts with passion, and excited about joining us. I see the hiring process as a mutual investment, where one can foster a culture of trust and respect. This approach builds strong peer relationships that nurture growth and passion. I believe in the power of diverse perspectives, as they drive innovative thinking. At Lucenia, this diversity of thought is fundamental, propelling us toward continuous improvement and groundbreaking solutions. We value team members who can work asynchronously and bring unique ideas to the table, ensuring our collective success and innovation.

What advice would you give to aspiring entrepreneurs who want to enter the tech industry, particularly in the field of search and AI?

My advice to aspiring tech entrepreneurs is to focus on solving real problems and helping customers save money. Create impactful solutions that offer tangible benefits and measurable results. Reflect on your own individual experiences to identify problems and leverage your unique background to solve them, considering how others might benefit as well. Success follows when you address genuine needs. Engage with others, gather knowledge, and approach problem-solving creatively and efficiently. Never give up, and ensure your efforts are driven by a clear, beneficial purpose.

——

As the conversation came to a close I listened as Dr. Knize painted a vivid mental picture of a barren land between the oceans of Elasticsearch and Amazon OpenSearch, symbolizing the neglected customers left in a drought. He envisioned Lucenia as the solution, providing the “Lego pieces” that empower users to apply search functionality to their applications, whether they choose to run it in the cloud or on their own hardware. This approach offers the freedom to pick the right tool for the job, allowing for efficient scaling up to the cloud when necessary and scaling down when it is not.

Finally, I asked about the role of the open-source community in Lucenia. Dr. Knize explained that as an Open Core Company, Lucenia is built on many different open-source projects like Apache Lucene and others. The company is committed to contributing back research and development within these communities, aiming to enhance search data structures and make them faster and more flexible for modern infrastructure. “We will open-source a lot of that glue,” he concluded, reaffirming Lucenia’s dedication to innovation and community collaboration.

Through this insightful dialogue, it became clear that Dr. Knize’s vision for Lucenia is driven by a commitment to simplicity, flexibility, and most importantly: user empowerment. This positions the company as a transformative force in the search industry.