Welcome To Our Blog

When scalability bites – Part 1

Adarsh EA
July 8, 2022

Introduction

We have a platform from which we offer hands-on labs for upskilling and learning needs. This SaaS solution from Nuvepro has existed for more than 5 years now and has been serving labs well. Recently we started seeing some strange behaviors with the lab states. A lab state can be Started, Stopped, Created, Deleted, etc.

The high-level architecture looks something like the one below.

On a particular day, the support team started seeing a bombardment of tickets with multiple issues with lab creation and many other operations like Start, Stop, etc. As we started debugging the root cause, it took us to a hidden world.

Problem – 1

We are using RabbitMQ for the messaging and updating of lab states across product lines. There is a message listener which reads the lab status changes and updates them in the database. Recently we had done a refinement to improve the RabbitMQ message processing by adding more listeners to read and process messages faster. The product is under load balancer, but there was only one listener. This arrangement was changed to have listeners for each node when it comes up.

After some investigations, we realized that multiple listeners could mess up the lab state since the sequence is important. For example: If a user does Start of a lab and a Stop immediately thereafter, as we have multiple listeners processing data, the Stop state change might get updated in the database first and then the Start state. This makes the lab state incorrect.

Solution

We removed listeners running in each node and reverted it to have a single instance that processes lab state change messages.

Problem – 2

The above corrective action handled the issue of mismatch of lab state, however, we started observing slowness in the update of lab status when the load is high (say 300+ concurrent labs). This issue was getting worse over a period and the engineering team was handed over the task of identifying the root cause and fixing it as it started affecting all our customers.

We started suspecting the RabbitMQ listener as the publishing of events seems to be happening correctly from the middleware platform. When we reviewed the message listener code at the frontend, the logic was straightforward, it just receives the message and checks for the availability of the lab, and then updates the status.

The next step was to see how long the query takes to update the database with the latest status. We could see that it was taking more than 1 minute to update the DB. This was verified manually from the SQL CLI and the result was the same.

Found the root cause

The table for the labs was not indexed in the frontend platform hence when the data became huge the query was taking a long time. This was blocking the lab states from getting updated on time. Adding an index to the table resolved it and all the lab status started getting updated correctly.

The above event was an eye-opener towards scalability issues and when we started growing rapidly, unexpected new issues started cropping up.

Watch this space for more….

Sign up for Newsletter

Our Latest Posts

GenAI Adoption Maturity: Bridging CTO Innovation and CIO Integration Through Skilling – Insights from Nuvepro’s COO

Generative AI (GenAI) is reshaping how organizations think about automation, creativity, and productivity. Yet, despite its promise, GenAI adoption remains fragmented – largely driven by CTO-led experimentation, with CIOs cautiously observing from the sidelines. The missing link? Skilling. Without a skilled workforce and a culture of responsible innovation, GenAI risks stalling before it reaches enterprise maturity. The GenAI Adoption Maturity Curve To understand the dynamics of GenAI adoption, we can visualize three overlapping trajectories: Skilling: The Strategic Enabler Skilling is not just a support function – it’s a strategic enabler that: Creating a Conducive Environment for Skilling To accelerate GenAI maturity, organizations must invest in: Skills Validation: The Fail-Safe for Enterprise Readiness Skilling alone isn’t enough – skills must be validated in real-life scenarios. This ensures: Real-world simulations, hands-on labs, and scenario-based assessments are essential to move from learning to readiness. Real-World Lessons from Early Failures Early adoption has shown that enthusiasm without structure can lead to missteps: These failures underscore the need for skilled, validated, and responsible adoption. Skilling as the Bridge – Enabled by Nuvepro GenAI’s journey from innovation to enterprise integration hinges not just on technology, but on capability building. Organizations must empower their teams to experiment responsibly, build confidently, and scale sustainably. This is where Nuvepro plays a pivotal role. With its hands-on skilling solutions, Nuvepro provides: By partnering with Nuvepro, enterprises can bridge the gap between CTO-led innovation and CIO-led transformation, ensuring GenAI adoption is not just fast – but also safe, scalable, and sustainable.

AI Agents Are Enterprise-Ready – But Most Teams Are Still in Training Mode

Agentic AI is ready to transform how work gets done – but most teams aren’t equipped to build AI Agents or deploy them. To move from hype to real impact, enterprises need AI-powered skilling built for project readiness. AI Is Everywhere – But Impact Isn’t In boardrooms, strategy decks, and LinkedIn posts alike, AI is the business buzzword of the decade. According to McKinsey’s 2024 AI adoption survey, over 80% of enterprises have integrated GenAI tools into at least one business function. Whether it’s content creation, customer support automation, or operational analytics, companies are eager to leverage AI’s potential. Yet, here’s the contradiction: Few discuss the fact that less than 15% of these organizations report measurable, enterprise-level ROI from their AI investments. This isn’t just a minor hiccup in tech adoption for Custom AI Assistants. It’s a fundamental operational and strategic challenge. Despite increased budgets, AI courses, and vendor partnerships, most companies remain stuck in pilot mode not knowing how to build AI Agents, unable to translate AI experiments into scalable, revenue-generating solutions. GenAI Adoption, ROI, and Market Impact (McKinsey Data Summary) Why the GenAI paradox? What’s Stopping GenAI from Scaling in the Enterprise? Why Aren’t More Teams Building AI Agents? While AI experimentation is widespread, few organizations have leaped to building and deploying AI agents at scale. This disconnect isn’t due to a lack of interest; it’s rooted in three persistent, structural barriers: How Nuvepro’s AI Project Readiness Platform Moves Enterprises Beyond Experimentation and more ROI? While generative AI and agentic AI tools continue to capture attention, most enterprises are still struggling to move from isolated pilot projects to scalable, production-ready AI agents that transform business workflows. The barriers are clear: a persistent skills gap, and no ROI in returns.  Nuvepro’s AI Project Readiness Platform is built to address these exact challenges, helping organizations operationalize AI initiatives faster, with greater confidence and measurable business outcomes. What Nuvepro Delivers Project Outcomes That Matter Nuvepro’s AI Project Readiness Platform is designed to deliver outcomes that go beyond learning metrics, directly impacting operational efficiency, project velocity, and the execution of enterprise AI strategy. Measurable Business Impact: 40% Faster AI Project Launch Skill-mapped, deployment-ready teams reduce project backlogs and accelerate time-to-market for AI-driven initiatives with the help of learning how to build Custom AI Assistants. Up to 40% Lower Operational Costs Workflow-specific AI agents automate high-volume tasks, reduce manual effort, and minimize SME dependency – unlocking operational savings at scale. 4-6 Weeks to Revenue Readiness Trained talent transitions from bench to billable roles within weeks, enabling faster client project onboarding and internal capability deployment. Margin Growth through Workforce Efficiency Achieve over 85% skill visibility, improving workforce planning and project staffing decisions. Cut SME evaluation time by 60% through automated, validated skill assessments aligned to enterprise KPIs. More Pilots, More Wins Confidently scale innovation programs and client-facing AI projects with validated, deployable teams, reducing project risk and increasing delivery success rates. The Core Pillars of Nuvepro’s AI Readiness Platform Why This Matters? AI agents won’t drive enterprise transformation through theoretical awareness alone. They require operational fluency, practical experience, and validated readiness to execute complex business workflows. Nuvepro enables organizations to scale their AI initiatives by closing the execution gap, building not just AI-literate teams but AI-proficient workforces capable of delivering measurable, business-aligned outcomes. Built for the AI-Driven Enterprise Nuvepro’s platform is architected for enterprise-scale AI adoption, addressing the full operational lifecycle from workforce readiness to production deployment, with enterprise-grade governance and system interoperability. Ready to Unlock Real AI ROI? Most enterprises today aren’t held back by a shortage of AI tools-they’re held back by a shortage of project-ready, validated talent capable of operationalizing those tools in business-critical workflows. Training alone isn’t enough. “To realize the full value of your AI investments, you need teams that can move from concept to deployment, delivering measurable outcomes against real business challenges”. Here’s how Nuvepro helps close that gap: It’s time to move from awareness to operational capability. From pilots to scalable AI outcomes. Your AI strategy demands a workforce equipped to build, deliver, and sustain AI initiatives, not just complete another course. Conclusion: AI-Powered Skilling for Project Readiness: From Hype to Real Business Impact – The Next Non-Negotiable Shift The AI conversation in enterprises has reached a pivotal moment. The numbers are clear, the case studies are real, and the market trajectory is undeniable. AI isn’t a question of “if” anymore – it’s a matter of “how well” and “how fast” organizations can operationalize it. And this is where most enterprises are falling short. Despite impressive adoption rates and a growing collection of GenAI tools, the business outcomes haven’t caught up. Productivity improvements and isolated pilot successes are no substitute for enterprise-level ROI, operational efficiency gains, and workflow transformation. The real value of AI – especially in its agentic form – lies in its ability to reshape decision-making, automate mission-critical processes, and enhance customer outcomes at scale. But achieving this requires a decisive, strategic shift. It demands more than AI awareness or one-off training initiatives. It demands project-ready teams equipped with applied skills, real-world experience, and validated operational fluency – ready to build, deploy, and sustain AI agents within complex enterprise environments. This is no longer a future-facing goal; it’s an immediate operational imperative. Organizations that continue to rely on theoretical learning and isolated experiments will inevitably fall behind, as competitors accelerate AI deployment in ways that directly impact profitability, customer retention, and market agility. The Path Forward Is Clear: Platforms like Nuvepro are no longer nice-to-have – they’re mission-critical.  Enterprises must equip themselves with infrastructure that not only trains their teams but also prepares them for real business problems, ensuring AI projects are deployable, scalable, and value-generating from day one. Agentic AI is ready to transform how work gets done. The question is – are your people? If your enterprise is serious about achieving AI-driven outcomes, it’s time to move beyond presentations and proof-of-concept demos. It’s time to build AI-proficient workforces that don’t just talk about transformation but actively deliver it. The AI skills

Why Skill Validation Is the Missing Link in today’s Training programs

In 2025, We’re Still Asking: Why Isn’t Learning Driving Performance? Billions are being spent. Thousands of training programs are being launched every year. Yet here we are—facing a truth that’s too loud to ignore: learning isn’t translating into performance. Let’s pause and reflect. Have you ever completed a training, proudly received a certificate, and still felt unprepared for the real challenges at work? You’re not alone. Despite major investments in learning platforms and certification programs, enterprises continue to face a fundamental challenge: turning learning into measurable capability. It is no longer sufficient to rely on a model where employees complete courses and organizations hope those skills translate into performance. This “train and hope” approach has crumbled in the face of increasing business complexity, fast-changing technologies, and pressure for real-time results. Enterprises today are navigating a growing disconnect—the widening gap between upskilling and actual job readiness. While the number of training programs has increased, so has the frustration among team leads and hiring managers who realize, often too late, that employees are not ready to perform the tasks they were trained for. This gap is not just a training issue; it is a business risk. According to Lighthouse Research & Advisory, only 16% of employees believe their skills are being developed for future success. This alarming figure comes despite organizations pouring record-breaking budgets into Learning & Development (L&D). So where’s the disconnect? Why is the gap between learning and doing still so wide? The High Cost of Skills Gaps The urgency of solving this issue cannot be overstated. According to current projections, 85 million jobs may go unfilled in the next few years due to a lack of skilled talent. The estimated cost of this shortfall is a staggering $8.5 trillion in lost revenue globally. This is not a distant scenario but a rapidly approaching reality. Surveys reveal that while a majority of organizations—around 83 percent—acknowledge having skills gaps, only 28 percent are taking effective steps to address them. The reasons behind this gap are complex, but three consistent challenges emerge across industries: visibility into real-time skill levels, mechanisms to validate whether learning has truly occurred, and the ability to act quickly based on skill readiness. This lack of visibility, validation, and velocity is limiting the return on learning investments. More importantly, it’s hindering business agility in a world where time-to-skill is critical. What Exactly is Skill Validation? Let’s be clear—Skill Validation is not a buzzword anymore. It’s not just a new checkbox in the L&D strategy document. It’s a paradigm shift—a change in how we approach talent development, assess readiness, and ensure that learning has real-world impact. For far too long, training programs have been measured by inputs: But the truth is, none of these guarantees job readiness. You can complete ten courses on cloud computing and still struggle to set up a basic cloud environment. You can ace a leadership development program and still falter when managing your first real team crisis. Why? Because completing training doesn’t always equal competence. Skill validation flips the narrative. Instead of asking: “Did they finish the course?” We ask: Can they do the task in a real situation, or Can the person actually do the job when put in an actual project? Skill validation helps in true learning by doing There is a massive difference between knowledge acquisition and skill validation. It’s real practice that shows whether someone is truly ready. Skill validation is not about learning in isolation—it’s about learning in context. It’s about immersing learners in real-life scenarios, simulated environments, and hands-on tasks that mirror the challenges they will face on the job. What Does Skill Validation Actually Look Like? Skill validation can take many forms, depending on the role, industry, and level of expertise. Like, for example, In every case, the individual is not just recalling information—they’re applying it. They’re making decisions, solving problems, and adapting in real time. This is the kind of learning that sticks. This is the kind of learning that builds confidence. And most importantly, this is the kind of learning that prepares people for the unpredictable nature of work. Skill validation is: It ensures your employees aren’t just trained—they’re trusted.. Why Skill Validation Is a Priority Now The rapid advancement of technologies such as artificial intelligence, cloud computing, DevOps, and cybersecurity tools has shortened the shelf life of technical skills. Job roles are evolving so quickly that the lag between training and application can result in irrelevance. Moreover, threats such as security breaches or project failures demand instant readiness from employees, not a six-month wait to assess post-training performance. In this context, relying solely on traditional learning models is no longer viable. Businesses need to know—immediately—whether a new hire is ready to deliver or whether an internal employee is prepared for the next level of responsibility. Skill validation addresses this need by offering evidence-based assurance of workforce capability. Being “almost ready” isn’t enough in today’s fast-paced business landscape. Organizations need people who can deliver from day one. Project timelines are tight, customer expectations are high, and there’s little room for error. This is why skill validation isn’t optional anymore—it’s essential. It ensures your training efforts aren’t just about checking boxes. It ensures your workforce is not only engaged but equipped. It bridges the final and most important gap: from learning to performing. Integrating Skill Validation Into the Learning Ecosystem For organizations aiming to embed skill validation into their talent strategies, the approach involves three key steps: Establishing Visibility: The first step is to identify current skill levels across roles. This requires tools that go beyond static self-assessments and instead gather real-time performance data from immersive, task-based activities. Embedding Validation in the Learning Journey: Skill validation should not be a post-training activity. It should be integrated throughout the learning process—from initial assessments to final evaluations. This ensures that learning is anchored in outcomes, not just content completion. Enabling Agility Through Continuous Feedback: With validated data on individual and team capabilities, organizations can respond faster—by tailoring interventions, accelerating project readiness, or rerouting resources

Welcome To Our Blog

When scalability bites – Part 1

GenAI Adoption Maturity: Bridging CTO Innovation and CIO Integration Through Skilling – Insights from Nuvepro’s COO

AI Agents Are Enterprise-Ready – But Most Teams Are Still in Training Mode

Why Skill Validation Is the Missing Link in today’s Training programs

Categories

Company

Partners

We Cater To

Legals

Resources