Friday, July 19, 2013

Eavesdropping at the call center


As performance engineers, we’re all focused on improving the responsiveness (throughput/response time) and scalability of the application in hand and hold a bucket full of tuning tricks to use at the right time. Do we really know what the users of the application feel about using the new system or does it meet the business requirements–well, we know about workload modeling for an application, would that be sufficient to achieve the business goal. These are some of the questions that we can definitely answer when we spend quality time with the user or be in user’s shoes.
As a performance consultant at one of our healthcare payor clients’ legacy modernization project, I wanted to understand the Customer Service Representative’s (CSR) perspective of the new application that has been rolled out. As in any typical legacy modernization project, members in the system are migrated in phases and the CSR’s get to use the old as well as the new* system at the same time. In the recent past, there has been anxiety on the IT side of the organization about the additional membership migrations planned in addition to the existing perception about the poor performance of the new system. To understand the problem first hand, I managed to secure some chair-side time at the call center and that was revealing
·         CSR’s do not use just the legacy and the modern application, but a host of 10 different applications and yes, there’s lot of opportunity for automation and usability engineering.
·         I also noticed approximately, 70% of the call handling time was spent on an application (letter server) that was not on the radar for the performance team.
3 hours spent at the call center also told me that the application that’s getting migrated was not as bad as it sounded
But wait. While I was looking for SPE (Software Performance Engineering) opportunities, I was also listening to the content of the calls and taking active notes. Mining through the notes at the end of the day provided me cues to more fundamental problems- one of the CSR’s was able to manage about 10 calls in 40 minutes and the 11th call lasted for about 30 minutes due to the very nature of the call and the caller. Pareto’s rule directs us to focus on the 11th call but a cursory look at the first 10 calls also signals an opportunity, all the 10 calls were inquiry calls about various services.
Here’s a rough listing of the calls and its content:
Enrollment
Billing
Claims
Post Sales Service
Inquiry calls
Can you confirm that my application has been accepted and I have coverage
Can you provide me the status of a claim
Can you send me a proof of coverage for the year 2012 to my new address
Non-inquiry calls
I am calling to make a payment for my dental plan as I am not able to pay online and the website says that the membership ID is invalid
I received a mail stating that my claim for my visit to the chiropractor has been denied.
I have been receiving mails from your company about claims denial, I am not even a member in any of your plans
Probable  causes (Detailed analysis required)
No active communication with customers regarding coverage start date
Incomplete website with no provisioning to pay for dental plans
Unclear or inconsistent communication
No active communication on the claim status.
Unclear and Incomplete communication about claim processing


As we would all agree about improving the call handling time of the 11th call via automation, application responsiveness, usability engineering and other techniques, we also see an opportunity to eliminate the first 10 calls and fix the underlying service problems- a true value to the business- a case where non-functional requirements driving functional requirement?

Saturday, July 7, 2012

Quotes

See it, Believe it, Act on it!!

Be the Change you want to be!

 I must be willing to give up who I am to become what I will be!

Fake it till you've become it!

Probability is a precise representation of uncertainity


Sunday, May 20, 2012

Capacity Planning





Performance tests
It is possible to achieve accurate performance prediction results by building a real environment with the expected number of users processing real applications using real data.

In the event of the non-availability of the production environment replica, a  set of vertical/horizontal  scalability tests will be carried out on a given application to determine the CPU/memory trends and this will not be linear. It has been proven that the polynomial/model can determined reliably if the environment can be mimicked up to 50%

When performance environment <50% of production

Linear Projection
A linear projection is anything from simple diagrams made with spreadsheets to more sophisticated methods. The accuracy bandwidth of projection tools is about 20% for utilization

CPU
·         CPU seconds per transaction =Number of CPU’s * CPU Utilization * 3600/Users*Transactions-per-hour-per-user
·         Estimated CPU Power = Users * Transactions-per-hour-per-user * CPU seconds per transaction/3600

Memory Sizing
·         Core system
o    AIX, daemons, and basic file systems
o    Uses a standard number
·         Per user memory
o    To run processes for users or batch tasks
o    Needs to be multiplied by the number of users in the system
o    Is best measured but can be guessed based on experience
·         Disks Cache
o    For database-like applications
o    Can be recommended by the vendor based on data volumes
o    Measured on other systems of similar data volumes
o    Estimated as a proportion of the data volumes
·         Application binary size
o    Should be measured
Disk Sizing

If disk I/O rates for the transaction are measured (either physical or logical), then the disk recommendation can be based on operations per second rather than purely on disk size. Disks have a known maximum operations per second value. This allows you to calculate the number of disks for a given workload.

Keep in mind that a bottleneck is not necessarily bad. While you will always have bottlenecks, and removing one creates another, removing a bottleneck improves performance. Other factors provide additional cushion.


Non-linear models
Analytic Models
Based on mathematical methods, such as the queuing theory, analytic methods can provide insight into forecasting CPU utilization, response time evaluation, capture ratios, effect of buffering, effect of queuing, and so forth. Based on measured data from today’s environment, various configurations and growth scenarios are studied and documented. Generally, analytic tools have an accuracy bandwidth between 8% to 15% for utilization. This depends on the scenario, level of detail, and provided data.


Discrete Models
A discrete method is an application of discrete simulation. Unlike analytic simulation, discrete simulation is not based on mathematical formulas. Using discrete capacity planning methods, dissimilar workloads and their effect on each other are modeled. Discrete simulation tools usually have an accuracy bandwidth of 5% to 10% for utilization. However, the accuracy may be more or less than 30% for response time. This depends on the scenario complexity and level of detail.

Must a model be perfectly accurate to be effective? Given the nature of modeling, the answer is No. The true value of modeling lies in its ability to effectively set expectations with the user community and management. It must explain what happened if there is a discrepancy between expectations and actual results. When planning decisions (selecting a new platform, acquiring more hardware, performance management and tuning decisions, etc.) are based on intuition, there is a high risk of performance surprises. Modeling helps to mitigate that risk.

Even when the results are not 100% accurate, the benefits of modeling are still tremendous:
  • Modeling can help you identify potential bottlenecks, evaluate different performance management and capacity planning alternatives, justify recommendations, and set realistic expectations.
  • Modeling reflects how sensitive performance is to proposed or expected workload and database size growth, scheduling changes, database and application tuning, or hardware configuration upgrades.
  • Performance prediction results provide a baseline for setting realistic expectations and improving the quality and effectiveness of proactive performance management.
  • Analysis of the trade-off and different alternatives presented are never perfect. However, models improve the level of understanding and communication between database administrators (DBAs), performance analysts, management, and users.
  • Models help to plan, manage, and control performance more effectively than intuition alone.

The following list classifies the good, bad and the ugly conditions from the utilization perspective
Good Bad and the Ugly

Monday, January 23, 2012


AIX CPU
Micropartitioning –Better utilization of CPU resource
Entitled Capacity—Guaranteed amount of Proccessing Units
Processing Units: Each partition is configured with a percentage of execution dispatch time for each 10 msec timeslice (dispatch window)
























Say EC=2.5
PU    Time with in dispatch window of 10 msec
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  1. 10
  2. 20
2.5    25 msecs


So 25 milliseconds will be divided between the 3 processors..8.x across each


But, the maximum PU's that's available for LPAR-1 is 3.0—30 milliseconds. This can happen only if the LPAR is uncapped.
Conferred calls: Virtual processor cycles of one processor is donated to another if it's waiting on the other's input
Simultaneous Multi-threading
    Enabled by default
    Can be disabled for scientific applications—number crunching
Each VP's in POWER 6 will use 2 logical CPU threads
Each VP's in POWER 7 will use 4 logical CPU threads
Maximum processor pools supported in POWER6 --64
Can be controlled by smtctl
Commands
Sar
Lparstat
Mpstat
Lsattr –E1 proc0
Power Hypersvisor schedules all the micropartitioning
Schedo command can be used for tuning hypervisor scheduler
VP_needed= physc+Vpm_xvcpus
Context switching:Too many processor can lead to lot of time spent on context switching
Virtual Processor folding


Dedicated Processor
Processor and memory affinity for best performance
Unused capacity lost
Less granular units of allocation
Can work in donating mode



Capacity entitlementàEntitled Pool capacityàMax Capacity
Virtual Processor Resolution Level
Level-0 resolution
If cycles required for an LPAR can be resolved within the same pool
Level-1 resolution
Available cycles available over and above the cycle requirement can be consumed by a different pool
Virtual Ethernet Adapters (Firmware)
LPARs communicate between each other using virtual ethernet adapters
(tagging of datagrams)
Virtual I/O Servers
When LPARS wants to communicate across the network
VLAN—priorities can be established
SEA-Shared Ethernet Adapters
SCSI—Storage
Threading/non-threading
Here threads are used for transferring datagrams
Virtual scsi and virtual Ethernet can communicate each other using threading
Threading hampers performance if used when SCSI is not there
When threading is enabled, a buffer is created that the kernel thread
When threading is not enabled, an interrupt is used for communication
Integrated Virtual Ethernet/Host Ethernet Adapter



Multi-core scaling –MCS


Virtual SCSI
Queue depth value should have the same as VIO server
N_Port_ID_Virtualization
LUN??


Partition mobility
Active memory sharing



Active memory expansion—works based on the memory compression set by the expansion factor. Can be monitored using lparstat Target memory expansion/target memory expansion size parameters
%xcpu: Percentage of CPU utilization used for AME compression and decompression activity
dxm: Memory deficit in MB
Command
Amepat
SMT
A logical processor has many registers namely, floating point, integer, load and save – they are not used at the same time, so using SMT enables the usage of the registers at the same time. If 2 registers have to be used at the same time then there is collision and there is a wait.


Process/Thread









WPARS
Application WPAR
System WPAR

 

Monday, December 26, 2011

Micro-partitioning

It's been a while since I blogged. I find it as a great outlet and store for my thoughts and readings.

With the advent of virtualization, CPU /memory/storage/os can all be virtualized--imagined and it makes things little convoluted to comprehend. I have seen many experienced folks getting confused with these concepts.

Here's my attempt to clarify and

Whole processor partitions by definition contain no fractional processor allocations and thus contain an integer number of processor whereas, micro-partition provides flexible and efficient use of system hardware resources by allowing physical processors to be shared (time-sliced) between micro-partitions.


In micro-partitioning,each LPAR can operate in 3 different modes:

Guranteed/Dedicated
  • LPARs are defined as dedicated or shared
  • Dedicated partitions use whole number of CPUs
Shared partitions use whole and/or fractions of CPUs (smallest increment is 0.1, can be greater than 1.0)
  • Borrowing/Using extra CPU cycles from the pool
  • Donating/Donate CPU cycles to the pool
Shared processor pool - subset (or all) of the physical CPUs in a system
Physical processors shared among all of the SPLPARs within the shared processor pool

Entitled processing capacity expressed in 0.1 CPU increments
  • Desired: Size of partition at boot time
  • Minimum: Partition will start will less than desired, but won’t start if minimum capacity not available DLPAR changes to desired cannot be below the minimum.
  • Maximum: DLPAR changes to desired cannot exceed this capacity
Uncapped:
  • No Processing unit usage is limited to desired setting and cannot access extra cycles from the pool
  • Yes Processing unit usage is allowed to exceed the desired processing unit setting but limited to a maximum of 1 for a Virtual processor


Use weighting to determine preference for spare cyclesAutomatic Load Balancing (default is 128, 0 implies no use of spare cycles, 255 is max Weight)




Virtualization is classified in different layers as
  • Physical/dedicated
  • Virtual/shared
  • Logical (if SMT is turned on)

Useful links


http://www.ibmsystemsmag.com/aix/administrator/lpar/An-LPAR-Review/?page=1


http://regions.cmg.org/regions/stlcmg/files/Download/Presentations_2009-05/CMG%20Processor%20Virtualization.ppt


http://www.ibmsystemsmag.com/aix/administrator/systemsmanagement/Configuring-Processor-Resources-for-System-p5-Shar/?page=1

http://regions.cmg.org/regions/phcmg/Feb08PWeilnau.ppt

Wednesday, August 10, 2011

Meaningful Performance Test

Non-linear:
Analogy-10 women cannot work together to get a baby in a month

Problem resolution:

Need profiling logs when the test is run with
Correct application and version
Usecase
Config
Database
Problematic issue must be reproduced

Scale is important
Water in a bucket and beach and their physics are different

Performance issues
-Performance Tips are for waiters, analysis for engineers
-Technicians read from the list and they become engineers when they add to the list

All CPU's WAIT at the same speed

Maserati has to wait in a STOP sign like a single threaded application, CPU works on the running code,not the waiting code!

Data quantity and quality plays a major role in test validity


Sunday, August 7, 2011

Trade-off's

I want to go to heaven, but I don't want to die

For a project the trade-off is--Fast, Good, Cheap.Pick Two!

Principles of optimization:

Our intuitions of the true costs of a program are usually wrong. Only by measuring can you confidence that you are having a positive effect on performance. When optimizing performance, it doesn't add value to speed up code that doesn't consume a significant execution time, measure first, back-out if it's not significant...don't waste time!


Key is to establish a balance between the browser and the server, AJAX provides an effective alternative to page replacement.