COST MANAGEMENT OF ENGINEERING PROJECTS

UNIT - 1

INTRODUCTION TO COSTING CONCEPTS

1. OBJECTIVES OF A COSTING SYSTEM

The objectives of a costing system are to: cost ascertainment, provide information for planning and
control, and provide information for decision making.

A costing system is a framework that helps companies estimate the costs of their products, services, and
other cost objects. It also helps with profitability analysis and inventory valuation. A costing system helps
collect, organize, and communicate information about costs, such as how much it costs to produce a
product or service, and which factors affect those costs.

Here are some steps to implement a standard costing system:

1. Set objectives

2. Determine cost standards

3. Establish a cost accounting system

4. Analyze and report variances

5. Review and update cost standards

Cost estimation is an important part of project management that helps stakeholders understand the
financial implications of their decisions. It can also help organizations make informed decisions about
whether to pursue a project or service, and how to allocate budgets for the best return on investment.

2. COST CONCEPTS IN DECISION-MAKING

Cost concepts in decision-making refer to the various types of costs and their implications when making
business decisions. Understanding and considering these costs is crucial for organizations to make
informed choices that maximize profits, minimize costs, and allocate resources efficiently. Some
important cost concepts used in decision-making include:

 Opportunity Cost: The value of the next best alternative that is forgone when a decision is made.
In other words, it’s the cost of not choosing the next best option. Considering opportunity
costs helps organizations make better choices by evaluating the trade-offs between different
courses of action.

 Incremental (Marginal) Cost: The additional cost incurred when producing one more unit of a
product or providing one more unit of a service. Incremental costs help decision-makers
understand how costs change with varying production levels and assess the profitability of
expanding or reducing production.

 Sunk Cost: Costs that have already been incurred and cannot be recovered, regardless of future
decisions. Sunk costs should not influence decision-making, as they are irrelevant to future
outcomes. Focusing on sunk costs can lead to irrational decisions, such as continuing a project or
investment solely because of the resources already invested, even if it’s not profitable going
forward.
Page 1
  Fixed and Variable Costs: Fixed costs are expenses that remain constant within a specific range
of activity or time, regardless of changes in production or sales volume. Variable costs change in
direct proportion to changes in production or sales volume. Understanding the relationship
between fixed and variable costs helps organizations make better decisions related to pricing,
production levels, and cost control.

 Direct and InDirect costs: Direct costs can be traced directly to a specific product or service,
while indirect costs cannot be directly traced and are allocated across multiple products or
services. Understanding the difference between direct and indirect costs helps organizations
allocate costs more accurately and make more informed decisions about pricing, resource
allocation, and cost control.

 Relevant and IrRelevant costs: Relevant costs are those that will be affected by a specific
decision and should be considered when making that decision. Irrelevant costs are those that will
not be affected by the decision and should not influence it. Differentiating between relevant
and irrelevant costs helps decision-makers focus on the costs that matter and ignore those that
don’t when evaluating various options.

i.RELEVANT COST

What is Relevant Cost?

Relevant cost is a management accounting term that describes avoidable costs incurred when making
specific business decisions. This concept is useful in eliminating unnecessary information that might
complicate the management’s decision-making process. Businesses use relevant costs in management
accounting to conclude whether a new decision is economical.

A particular cost may be relevant for one situation but irrelevant for another. The opposite of relevant
costs is sunk cost or irrelevant costs, which refers to the expenses already incurred. Thus, incurring
an expense may be avoided by deciding not to perform a certain activity.

How Relevant Cost is used in Decision Making?

The three main types of relevant cost examples considered during a business decision are:

 Whether to make or buy.

 Close a business unit or continue production.

 Special orders.

#1 – Make or Buy

A company that deals with making finished goods requires specific parts. The company has to decide
whether to make the parts internally or outsource. Naturally, the lowest cost alternative is the
Page 2
best. Direct materials, direct labor, and various overhead costs are examples of the make or buy
situation.

Suppose a company wants a part of some machine. They can buy the part from a vendor or make it in the
factory. The company shall free some space that can be leased if it decides to outsource. The management
can outsource to make an extra income from leased space. The relevant cost analysis thus helped the
company to conclude that buying the part was a more financially sound decision.

For example;

XYZ Company manufactures motor vehicle spare parts that need a specific piece of equipment.
Purchasing from a supplier costs $5 per unit. But the company can make the same piece internally as
well. The company requires 50,000 units of spare parts per annum. By producing internally, the company
incurs the following costs:

Direct materials=$2/unit

Direct labor=$4/unit

Overhead costs=$1/unit

Special tools=$40,000

According to the above illustration, it will cost XYZ $250,000 to buy from a supplier. And it will cost
$390,000 to make the same internally. Therefore, XYZ should continue outsourcing.

#2 – Continue Production or Close Business Unit

A major dilemma regarding any business at some point is whether to continue operation or close business
units. Here, the management needs to consider whether the units are making expected income or have
high maintenance costs. Appropriate cost analysis form plays a primary role in making that decision.

For example;

The company Billy’s makes cheese worth $10,000 per month. Maintenance cost for machinery is $3,000,
$2,000 for material, $2,500 for labor, and $1,500 for miscellaneous costs. Overall expenses amount to an
income of $10,000. So, the Billy’s might think of discontinuing the cheese unit. Billy’s might continue
with cheese production if the expenses are lower, like $ 7,500.

#3 – Special Orders

In business, a customer may request a one-time item from a company. They could have made this order
right after the company had calculated all its costs on normal sales. The company shall then consider the
lowest price for producing that order. It considers taking special orders if the costs involved will generate
income in the long run.

Page 3
Before accepting special orders, the company must put into consideration;

 If it has the necessary capacity to complete the order.

 If it has already covered the cost of production.

 If it has analysed all the fixed costs

If the product cost price is below production cost, the company can safely decide to take special orders.

Types of Relevant Costs

There are four types of relevant costs;

 Avoidable costs

 Incremental costs

 Opportunity costs

 Future cash flows

#1 – Avoidable Costs

The term is also called variable costs. If a company decides not to undertake an activity, the company
can avoid some expenses.

It happens when the company opt-out of other activities that can save it from incurring expenses. Variable
costs vary with different levels of production. It means that if there is zero production, there is no
spending.

Variable costs=Quantity output x Variable cost per unit output

#2 – Incremental Costs

Along the line of business, there is the production of several units. These additional units have a price tag.
Thus, these costs increase as the production increases or drops with low production. They are called
incremental costs.

Along the line of business, there is the production of several units. These additional units have a price tag.
Thus, these costs increase as the production increases or drops with low production. They are called
incremental costs.

#3 – Opportunity Costs

Page 4
Opportunity costs are associated with choosing between two alternative options. The loss of benefit due
to an alternative option is the opportunity cost, also known as the alternative cost.

For example, a person has to choose between vacationing and spending time with their family. In this
context, opportunity cost is the cost of the holiday and visiting new places if the person decides to go on
vacation rather than stay home.

#4 – Future Cash Flows

The future expenses that might occur due to a decision made in the present are called future cash flows.
The current value is used to project future revenues to see if a decision will incur future costs. Here, we
can price the expected ongoing-project revenues with the current value. Then, a discounted rate is
formulated to arrive at discounted cash flows.

ii. DIFFERENTIAL COST

Differential Cost Definition

Differential cost is a technique of decision-making in which the cost between various alternatives is
compared and contrasted with choosing between the most competing alternative. It is useful when you
want to understand a) Whether to process the product further or not and b) Whether to accept an
additional order at a lower current price.

It differs from the marginal cost because marginal cost includes labor, direct expenses, and variable
overheads, whereas differential cost includes both fixed and variable costs.

Examples of Differential Cost

The following are examples to understand this concept in a better manner.

Example #1

ABC Ltd is a company that produces card boxes. Its monthly cost statistics are as follows:

· Units made and sold: 800 units per month

· Maximum production and sales capacity: 1200 units per month

· Selling price: $30

The bifurcation of cost is as given below:

They have an alternative to increasing the production of up to 900 by reducing the selling price to 28.

Please evaluate the feasibility of the option.

Solution
Page 5
Option 1: Present situation: selling price 30

Hence, the entity is earning a profit of $5,600 per month.

Option 2: Alternative to increase the production

Based on the two options, the cost of both the options can be evaluated as given below:

From the above analysis, we can observe that with the change in the alternative, an entity will have to
incur an additional cost of $1,000. Hence, an increase in production is not advisable.

Usage of Differential Cost Analysis

 Getting Prices of Products: The optimum price quoted can be won.

 Accepting or Rejecting Special Orders: Whether to work out in any additional specific order in
business.

Page 6
  Adding or Eliminating Products, Segments, or Customers: Whether to continue or diversify
from any specific business segment or not.

 Processing or Selling Joint Products: Whether to co-produce or co-sell the products or jointly
market the products;

 Deciding whether to Make Products or Buy them: Whether to manufacture the product or
leverage the production facility of others.

Accounting Treatment of Differential Costing

The differential costs can be fixed, variable, or semi-variable costs. Users leverage the costs to evaluate
options to make strategic decisions positively impacting the company. Hence, no accounting entry is
needed for this cost as no actual transactions are undertaken, and this is the only evaluation of
alternatives. Also, no accounting standards can guide the treatment of differential costing.

Conclusion

Thus, differential cost includes fixed and semi-variable expenses. It is the difference between the total
cost of the two alternatives. Therefore, its analysis focuses on cash flows, whether it is getting enhanced
or not. Therefore, all variable costs are not part of the differential cost and are considered only on a case-
to-case basis.

iii. INCREMENTAL COST

What Is Incremental Cost?

Incremental costs are the costs linked with the production of one extra unit, and it considers only those
costs that tend to change with the outcomes of a particular decision. In contrast, the remaining costs are
deemed irrelevant.

Incremental Cost Explained

The term incremental cost refers to the cost that the business incurs for producing an extra unit. It is a
process where the companies take into consideration different expenditures that are involved in the
production process like procurement of raw materials, labor cost, packing and transportation or finished
goods, utilities like electricity, etc.

This concept of incremental cost of capital is useful while identifying costs that are to be minimized or
controlled and also the level of production that can generate revenue more than return. The moment one
extra unit produced does not generate the required return, the business needs to modify its production
process.

Page 7
Organizations can do this analysis for incremental costs of obtaining a contract, in order to determine
whether the concerned business segment is able to generate targeted profit and keep track of incremental
revenue, always being more than the cost. It provides guidance regarding decision-making for the
management in terms of pricing, allocation of resources, planning or production quantity, sales target,
profit target, etc.

Thus, we see that factors taken into consideration in this concept are those that change with production
volume. The fixed costs are not considered over here because they remain the same. at all production
levels. For this reason, it is also known as marginal cost.

Formula

The standard method or formula for incremental costs of obtaining a contract, that a business will use
to calculate this cost is as follows:

Incremental Cost = Total cost incurred for increased production level – Total cost incurred in
previous production level.

In the above formula, the total cost of increased production refers to the previous volume and the new
units added to it. From this value, the cost of volume is deducted to get the cost. However, none of it will
include the fixed costs since they will not change due to volume fluctuation.

Example

Let’s take an example to understand the incremental cost of capital better:

Assuming a manufacturing company, ABC Ltd. has a production unit where the cost incurred in making
100 units of a product X is ₹ 2,000. The company wants to add another product, ‘Y,’ for which it incurs
some cost in terms of salary to the additional labor force, raw materials, and assuming that there was no
machinery, equipment, etc., added.

Let’s suppose now, after adding the new product line, it can produce 200 units at ₹ 3500, so here the
incremental cost is ₹ 1,500

Like in the above example, it is evident that the per-unit cost of manufacturing the products has decreased
from ₹ 20 to ₹ 17.5 after introducing the new product line. However, this may not be true in all cases.
Identifying such costs is very important for companies as it helps them decide whether the additional cost
is in their best interest.

It is not necessary that such costs can only be variable. Even fixed costs can contribute to the incremental
cost, for example, if there is a requirement for new machinery for adding the new product line ‘Y.’

Allocation of Incremental Costs

The basic method of allocation of incremental cost in economics is to assign a primary user and the
additional or incremental user of the total cost.

Page 8
If we look at our above example, the primary user is product ‘X’ which was already being manufactured
at the plant and utilizing the machinery and equipment. The new product only added some extra cost to
define ‘X’ as the primary user and ‘Y’ as the incremental user.

In the absence of any new product or any additional unit, the total cost that ABC Ltd. incurred while
manufacturing only ‘X’ is ₹2,000, so we’ll allocate this cost to X,

The additional cost of ₹ 1,500, incurred only to introduce the new product, will be allocated to ‘Y.’

This allocation can even change in the future course of business of ABC Ltd. when supposedly, if it
chooses to drop product ‘X,’ then product ‘Y’ or any other product might become the primary user of the
cost.

Incremental costs are also associated with the changes in the product’s pricing. Let’s suppose if the
overall cost per unit of a product is also increasing by incurring such cost, then the company would want
to change the price of the product to maintain or increase the profit. The incremental cost in
economics might work in or against the favor of the company. Such companies are said to
have diseconomies of scale, i.e., they have already reached the maximum limit of production volume.

But if the per-unit cost or average cost is decreasing by incurring the incremental cost, the company
might be able to reduce the price of the product and enjoy selling more units. Such companies are said to
have economies of scale, whereby there is some scope available to optimize the utility of production.

Considering that the price of each unit of product ‘X’ is ₹ 25, the profit initially was

Net Profit = ₹ 500

Also, considering after introducing the new product line, the price for both ‘X’ and ‘Y’ is kept at ₹ 25,
the profit here will be:

 Net Profit =(200 X 25) – (200 X 17.5)

 Net Profit = ₹ 1500

Page 9
To increase the sales to gain more market share, the company can leverage the lower cost per unit of the
product to lower the price from ₹ 25 and sell more units at a lower price.

Benefits

Some benefits of this type of cost are as follows:

 Resource allocation – This process helps in resource allocation for the business. Resources are
never unlimited. There is always a limitation to the amount of resource that can be procured and
applied in the production process. It is necessary to use them to their optimum capacity without
wastage. Incremental cost analysis ensures that resources are used wherever it is needed. Rise in
this cost means resources are being channalized in wrong direction.

 Plan production levels– Production planning is extremely important for every business so that
goods and services are available on time and as per demand in the market, with extra stocking up
of finished goods and services which not only occupies space but increase cost and lead to
wastage. Analysis of incremental or marginal cost helps in this process.

 Decision making– The entire production and planning process require decision making regarding
price, volume, sales, revenue target, demnd and supply analysis, etc. Analysis of incremental or
marginal cost contributes to this.

 Budgeting– Budgeting is of huge importance for every business process, which requires fund
allocation for smooth working. Cost analysis regarding extra production helps in getting an idea
about how much budget needs to be allocated so that revenue and sales increases leading to better
profitability.

 Profit target– Every company needs to set their own profit target for different departments,
which requires planning for increase in output and how much level of production volume will
actually be profitable for the enterprize.

 Cost control – The analysis of incremental or marginal cost involves identifying the areas that
will involve higher cost during production and either minimize them or optimize them. This helps
in surviving the competition and improving the efficiency of the process.

 Risk minimization– This concept helps in minimizing the risk associate with expansion of
production volume which is useful while launching new or improved goods and services in the
market. It identifies the addition funds that has to be invested and design useful risk management
practices.

Thus, the above are some benefits that the procedure of marginal cost analysis contributes to the entire
manufacturing process.

Incremental Costs Vs Margin Costs

Page 10
Incremental costs are also referred to as marginal costs, but there are some basic differences between
them.

 Incremental costs are mostly associated with choices or decisions and therefore include only those
additional costs caused due to the decision made; for example, it does not consider the cost of
machinery or equipment which was already there in the production unit, which is also referred to
as sunk cost because these costs will remain regardless of any decision.

 On the other hand, Marginal cost specifically takes into account the increase in cost for producing
one additional unit. It is often used to optimize production, while the incremental cost is not an
optimization tool.

Incremental Cost Vs Incremental Revenue

As the name suggests, both are meant to calculate the cost and revenue for extra or addition production of
goods and services. Let us look at the differences between them.

 The incremental cost analysis deals with analysis of cost for producing extra output and the latter
deals with the analysis of revenue earned for producing extra output.

 The former calculates the how much more cost will be incurred for increase in the output unit and
the latter does the same with the revenue.

 In the field of decision making, the incremental cost analysis suggests that if the analysis shows
increase in cost, that production should not be accepted, whereas the latter suggests that if the
analysis shows increase in revenue, such a production should readily be accepted.

 The aim of the business should be to minimise the former, whereas the aim of the business should
be maximise the latter.

Thus, from the differences given above we can understand that both the concepts have a lot of differences
but are equally useful for the organization for the purpose of production planning and ensuring growth
and expansion.

Companies can broadly use the incremental cost to analyze whether to produce the new product line in
the house or to outsource it and whether to accept a one-off high volume order from the customer or
business partner.

iv.OPPORTUNITY COST

What Is Opportunity Cost?

Opportunity cost is a term that describes the potential benefit one foregoes while choosing an alternative
over the next-best choice. They can be thought of as a trade-off. When one choice is chosen over another,
trade-offs occur in the decision-making process and represent the cost involved.

Page 11
It is the value a company loses when choosing between two or more alternatives. Therefore, a business or
individual must be aware of the opportunities they lose whenever they pick one investment or decision
over another. This requires careful weighing of the alternatives and deciding what’s best.

Opportunity Cost Theory Explained

Opportunity cost is the potential gains forfeited when a person, company, or investor selects one
alternative over another. One can very easily overlook the costs since they are not visible. The core
element of conventional economics is that demands are unlimited. The more the number of demands, the
better it is for the economy‘s growth. This is the foundation of this concept. Customers who purchase a
certain good or service miss out on the chance to benefit from something else’s utility.

In simple terms, an expense happens when an entity loses a chance or opportunity. Another related
concept in economics is the increasing opportunity costs. The law of increasing opportunity cost states
that if there is an increase in the production of one product, the opportunity cost to produce the additional
good will also increase.

Businesses may need to analyze the Opportunity Cost of capital, investments, etc. Similarly, investors
look into the cost of investing in one stock over the other. Opportunity cost, from the standpoint of the
investor, implies that the choices taken about investments will result in future gains or losses that are
immediate.

At its foundation, economics is the study of how people make choices in the actual world regarding the
distribution of limited resources. If there were no resources, it would eliminate the need to forgo any
preferred activity, allowing for completing all tasks. Therefore, every action has an opportunity cost.
Thus, it’s critical to make an effort to comprehend the full scope of these expenses before making
decisions.

Assumptions

Although this is an abstract and quantifiable term, it cannot be quantified, like the price of a commodity
or service sold or the cost of a good or service produced. Three key assumptions form the foundation of
opportunity cost for the consumer:

1. Wants are unlimited.

2. Wants and necessities are equivalent.

3. The consumer constantly pursues maximum personal net advantage.

The opportunity cost for the producer is based on the ideas that (a) the choice foregone is feasible and (b),
like the consumer, the producer continuously pursues maximum personal advantage.

The equation is as follows:

Opportunity Cost = FO (return on the best-forgone choice) – CO (return on the chosen option).

The difference between the projected returns from each choice serves as the basis for calculating
opportunity cost.

Types

There are two types of opportunity costs:

Page 12
#1 – Explicit Costs

Explicit costs are direct costs which associate with an action. They are easily identifiable as out-of-
pocket expenses. For example, payments made to employees, land and infrastructure costs, cost of
capital, operations, maintenance costs such as wages and rent, etc., belong to this category.

#2 – Implicit Costs

Implicit costs are implied costs that one cannot easily identify. They are the costs of firms utilizing
resources they could have used for other purposes. They correspond to intangibles and are not visible.
One cannot, therefore, report them easily. This can include a small business owner starting without a
salary to increase the company’s profitability.

Example

Dan has two options to invest in his monthly paycheck. Company ABC gives a rate of interest of 10%,
and company XYZ gives a rate of interest of 12%. However, Dan aims to earn some passive income.
Even though it shows only 10% interest, ABC has a track record of continuously providing dividends. In
contrast, XYZ has long gaps between its dividends. Therefore, Dan inclines more towards ABC, as he
sees it as the one that can provide a potential continuous source of income. In this case, the 2% rate of
interest he forgoes is his Opportunity Cost. (12%-10%).

Importance

The importance of opportunity costs can be summarized as the following key points:

 When making decisions like forgoing short-term gains in favor of a longer-term investment, these
costs help weigh each option’s benefits and drawbacks.

 These expenses help evaluate the viability of a specific business opportunity. It may also assist
individuals in deciding whether a particular course of action would benefit them in the long run.

 Making a sound decision from various options may become easier on knowing the cost of
forgoing and can make the process simpler.

 It helps determine the benefits and disadvantages of the present course of action or decision.

 Finally, It is beneficial while analyzing long-term objectives.

Limitations

Some of the limitations of opportunity costs are:

 Calculating these costs cannot predict future results accurately.

 Quantitative comparisons between the two available options can sometimes be challenging.

 These costs differ from individual to individual and circumstance to circumstance.

Opportunity Cost vs Trade Off vs Sunk Cost

Sunk costs are also referred to as historical costs, which have been incurred already and cannot be
recovered in the books. As sunk costs have already been incurred, they tend to remain unchanged and
should not influence future actions or decisions regarding benefits and costs. Opportunity costs are those

Page 13
that are forgone for the next best alternative. On the other hand, when individuals or a group of
individuals make decisions to endure sacrificing one thing to obtain more of another, this is a trade-off.

3. CREATION OF A DATABASE FOR OPERATIONAL CONTROL.

Operational databases are the digital workhorses that handle the constant influx of data that our modern
world generates. From tracking warehouse inventory to managing customer orders for a global
eCommerce giant, operational database examples and use cases are endless.

When it comes to getting the most out of operational databases, you should know the challenges they
pose. Many companies run into these issues and often get creative in finding workarounds. Now the smart
move is to avoid reinventing the wheel and take a page from others' experiences.

To make your data management journey even smoother, we've put together this handy guide. It will give
you a head start in understanding these operational database challenges and how others have tackled
them.

What Is An Operational Database?

An operational database, also known as a transactional database or OLTP (Online Transaction

Processing) database, is a database management system designed to efficiently manage an
organization’s day-to-day transactional operations. These operations typically include the creation,
retrieval, updating, and deletion of data.

Operational databases are typically used to support core business activities like order processing,
inventory management, customer relationship management (CRM), and more. Unlike a data warehouse,
which is made for decision-making and analysis, the operational database can efficiently store data
with all the crucial details like customer data, product listings, sales records, and other important data.

Operational databases are refreshed in real-time so you can instantly access the most updated data
available. These databases work really well in systems where fast and accurate data retrieval is
important, like in eCommerce platforms or banking systems.

Here’s what makes them special:

 Stability: They can accommodate distributed systems in their infrastructure. So even if one part
faces issues, the entire system continues to operate smoothly.

 IoT Features: The operational database system connects well with the Internet of Things (IoT),
always watching, checking, and offering its own solutions.

 Growth Support: Modern operational database systems can adjust and grow as businesses need.
They're fast and can manage many users at the same time.

Page 14
  Quick Insights: Operational databases provide immediate feedback to help make better choices.
It can mix in other tools to improve its feedback based on what the user wants, without changing
the main database.

5 Key Features Of Operational Databases

 Real-time Processing: Operational databases are ideal for real-time or near-real-time data
processing. They provide immediate responses to user requests which makes them a great fit for
applications that need quick access to transactional data.

 Normalized Data Structure: They use a normalized data structure to minimize data redundancy
and maintain data integrity. This includes organizing data into smaller, related tables and
establishing relationships between them using keys (e.g., primary keys and foreign keys).

 Concurrent Access: Operational databases support concurrent access by multiple users or

processes. This means that multiple users can interact with the database simultaneously and the
database system ensures data consistency and integrity through mechanisms like locking and
transaction isolation levels.

 High Transaction Volume: Operational databases can handle a high transaction. These
transactions can include inserting, updating, deleting, and retrieving individual records or small
sets of records. They are optimized for rapid, short-duration transactions and are capable of
processing numerous transactions per second.

 Data Consistency and ACID Compliance: Operational databases adhere to the ACID
(Atomicity, Consistency, Isolation, Durability) properties to maintain data integrity and reliability,
even when the system fails. These properties guarantee that database transactions are executed in
a way that ensures data consistency and prevents data anomalies.

Challenges & Solutions In Operational Database Management: 5 Examples To Consider

In these database operational management systems, every day brings new challenges and with those
challenges come opportunities for innovative solutions. Let’s take a look at 5 real-world examples that
show the hurdles these organizations faced in managing their operational databases and the creative ways
they found to overcome them.

Example Problem 1: Scalability Issues

Page 15
As businesses grow, their operational databases face a tough job: managing increased data volumes
and serving ever-growing user requests. When databases can’t keep up with these, users experience
delayed response times, potential outages, and sometimes even data losses.

Uber, the global ride-sharing giant, is a prime example. With their growth, they needed real-time data
processing to connect drivers and passengers, adjust pricing based on demand, and analyze metrics
related to riders, drivers, trips, and finances.

However, their existing database system had latency issues, with data ingestion taking anywhere
from 1.5 seconds to 3 minutes, depending on the source. The company, which heavily relied on real-time
data, found this inconsistency in data processing to be a major issue.

Proposed Solution: A Distributed Database System

Uber turned to SingleStore, a distributed database system, to address their scalability challenges.
SingleStore's design uses a massively parallel processing architecture that lets it run on multiple
machines.

This setup not only supports faster data ingestion but also combines new and existing information to
provide quick answers to queries. For Uber, this meant that they could make real-time decisions based
on analytical results and respond quickly to changing market dynamics.

Here’s how it works:

 Load Distribution: Spreading user requests across servers means no single server gets
overwhelmed.

 Data Redundancy: Storing data across multiple servers boosts data availability and resilience. If
one server fails, another picks up the slack.

 Enhanced Performance: With the right setup, data retrieval becomes faster as the system can
pull data from multiple sources simultaneously.

Example Problem 2: Data Integration Difficulties

Operational databases often need to interact with other databases or applications. However, the
process isn't always smooth. Mismatched data formats, disparate systems, or even different update
frequencies can all complicate integration. This difficulty causes incorrect data merges, loss of data, or
even system downtimes.

For example, a global bank wanted to comply with Anti-Money Laundering (AML) transaction

Page 16
monitoring and the Financial Conduct Authority (FCA). However, they hit a roadblock. The bank used
different systems, from mainframes to cloud platforms. Integrating data from all these systems into
their central data lake was a challenge, especially with limited internal resources.

Proposed Solution: Middleware Solutions & Data Mapping Techniques

Banks can use middleware solutions to handle data integration challenges. Middleware acts as an
important connector that bridges different systems for a smooth data flow. For a global bank dealing
with different systems, middleware can seamlessly link them to ensure data consistency.

Coupled with data mapping, it helps align data from different sources correctly in the database. This
combined approach not only makes the bank's data processes more efficient but also guarantees that it
has precise and comprehensive data to support its Anti-Money Laundering efforts.

 Seamless Integration: Using middleware and data mapping, banks can merge disparate databases
without loss or misinterpretation of data.

 Operational Efficiency: Smooth data flow between integrated systems means businesses can
function more efficiently and avoid downtime or data retrieval delays.

 Cost Savings: While there might be an initial investment in integration tools or expertise, in the
long run, it eliminates the costs associated with manual data handling or system errors.

Estuary Flow offers a specialized solution for operational databases. It understands the complexities of
integrating data from different sources. So if you are facing challenges with mismatched data formats
and system disparities, Flow is ideal for your business.

Estuary Flow prioritizes real-time data movement and transformation so even when data comes from
different systems, it remains consistent and up-to-date. This way, it directly addresses the issues of
incorrect data merges and potential system downtimes.

Example Problem 3: Security Vulnerabilities

Page 17
Operational databases, being at the core of business operations, are attractive targets for malicious
actors. Database security weaknesses can result in unauthorized access, data tampering, or even data
theft. These breaches can not only cause financial damages but chip away at the trust and value of a
brand.

Let’s take a look at Medical Informatics Engineering’s (MIE) example, a prominent developer of
electronic medical record software. MIE's database stored important information like names, social
security numbers, medical conditions, and more.

Because of a lapse in their security measures, an outsider accessed and exposed a vast amount of this
data on the internet. The security breach drew a lot of media coverage, led to multiple legal actions, and
dealt a severe financial blow to MIE.

Proposed Solution: Multi-Layered Security Protocols & Regular Audits

Multi-layered security protocols involve a combination of firewalls, encryption, user authentication

methods, and role-based access controls. Such protocols guarantee that only authorized individuals
access the database and that the data transmitted is encrypted.

Periodic security audits help identify and rectify vulnerabilities. Automated tools are used to monitor
database activity and flag any unusual actions for review. Let’s see some of the advantages of
implementing robust security measures.

 Trust and Compliance: Enhanced security measures build customer trust and help businesses
comply with data protection regulations.

 Protection of Sensitive Data: With strong security measures in place, sensitive information
remains out of reach from unauthorized entities.

 Avoid Financial and Reputational Damage: When companies protect their financial data
against breaches, they can avert potential financial losses and protect their reputation.

Example Problem 4: Poor Data Quality

Data quality serves as the foundation of every functional database. If the data isn't right, everything
built on it can crumble. Wrong data equals bad choices, inefficiencies, and lost money.

Page 18
Sectors like healthcare need accuracy and if you have unreliable data stored in your operational
database, it can be devastating. For example, in the U.S. healthcare system, misidentification is more
than just an administrative hiccup, and mismatched patient data has become a critical concern.

Beyond the obvious health risks, the economic consequences are also severe. Patient misidentification
results in about 35% of insurance claims getting denied. This discrepancy can then hit hospitals hard,
with an annual financial loss of up to $1.2 million for an average hospital.

Another major issue of data quality is that of duplicate records. It's not uncommon for a patient
to discover multiple records under their name in a hospital's database. With hospitals reporting an 8%
to 12% rate of duplicate records, the situation is concerning.

Proposed Solution: Data Quality Monitoring & Regular Data Audits

Data quality monitoring uses tools and processes to continuously check and validate the data for
accuracy, consistency, and completeness. This makes sure that the data entering the operational
database meets the required quality standards.

Regular data audits involve periodically reviewing the data to identify any inaccuracies or
inconsistencies. This helps rectify any errors and ensures that the data stays reliable over time. Here are
the benefits of focusing on data quality:

 Operational Efficiency: High-quality data reduces the chances of errors in operations for smooth
business processes.

 Informed Decision Making: Accurate data ensures that business decisions are based on reliable
information.

 Financial Savings: When businesses fix poor data quality issues, they can prevent financial
losses arising from misguided decisions or operational inefficiencies.

Example Problem 5: Latency Issues In Real-time Data Updates

Operational databases are expected to provide data on demand, instantly. When they can't, it's called
a real-time data access delay. This lag can hinder business processes, especially when milliseconds
matter. There can be many reasons for slow access speeds – overloaded systems, inefficient data
structures, or network issues.

Consider Coin Metrics – a company that provides important financial data. They collect data from
different sources and deliver it to their users in real time. One of their primary tools for this is the
WebSocket API which streams data live. However, this tool can be glitchy, especially during high
market activity.

These interruptions were a big problem for Coin Metrics which relied on constant data flow. Users
needed up-to-date data. Any delay or latency meant that decisions could be based on slightly outdated
data, posing a risk of financial setbacks for the traders.

Proposed Solution: Data Caching & Database Optimization

Data caching plays a major role in improving system performance. When frequently accessed data resides
in 'cache' memory, the system bypasses the longer process of fetching it from the primary database for
swifter data retrieval.

Page 19
Similarly, optimizing the database improves efficiency. The redundant data is eliminated through
effective indexing and refining queries which reduces the access times for data. When these strategies
work together, the database experience becomes smoother and more responsive.

 Immediate Data Retrieval: Caching provides rapid responses to frequent data requests and
reduces wait times.

 Consistent User Experience: With data delays minimized, traders can trust the platform to
provide accurate, timely information, solidifying brand reliability.

 Optimal Resource Use: With a superior query processing system, the database processes requests
using fewer resources, thus serving more users efficiently.

How Does Estuary Flow Help Tackle Operational Database Challenges?

Operational databases play an important role in many businesses. But as they scale, they run into
challenges like data movement, transformation, and extensibility. Estuary Flow is our DataOps
platform designed to streamline these processes and maintain the efficiency and reliability of your
databases. It comes with different useful features to tackle typical operational database problems.

Estuary Flow captures data from various sources and ensures real-time Change Data Capture (CDC) from
databases. With it, you can access the most current data. Flow is versatile when it comes to data
processing since it supports both streaming SQL and Javascript, letting you handle transformations in
different ways.

Key Features Of Estuary Flow

 Extensibility & Adaptability: Lets you easily add connectors and guarantees transactional
consistency for accurate data views.

 Reporting & Data Structuring: Offers real-time data flow reports and converts unstructured
data into structured formats for easier data management.

 Reliability & Survivability: Comes with built-in testing for data accuracy and features like cross-
region and data center support to prevent disruptions from external issues.

 Efficiency & Scalability: Uses low-impact CDC for efficient data use and is designed to manage
large data volumes, allowing businesses to grow without facing data management challenges.

 Data Capture & Real-time Updates: Efficiently captures data from various sources, like
databases and SaaS applications, and ensures real-time Change Data Capture (CDC) for timely
information access.

 latency views across systems, making sure the data remains current and easily accessible.
Page 20