Thursday, February 7, 2019

Monitoring Java Application Running In Cloud Foundry (Platform-as-a-Service)

This article defines the approach to monitor Cloud Foundry Application via any Java Visualization Monitoring tool like JVisualVM or JConsole.

This approach will enable monitoring of Java Virtual Machine utilization (like GC, CPU, Memory etc). Also, it can monitor threads and perform CPU & Memory Profiling.

Step 1: 
Add below parameters in the Cloud Foundry Environment variables for the Cloud Foundry Application to enable Remote JMX monitoring.
JAVA_OPTS: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9480 -Dcom.sun.management.jmxremote.rmi.port=9480 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
Restage the application to take the above parameters into affect.
    cf restage

Step 2:
Do local port forwarding for your Cloud Foundry application for JMX port running on Cloud Foundry Org/Space.
cf ssh -L 9480:localhost:9480

Step 3:
Use JVisualVM to monitor the Cloud Foundry Application.

CPU/Memory Profiling can also be enabled from JVisualVM          

Sunday, July 30, 2017

Learning From Performance Testing/Engineering:


ConcurrentModificationException occurs due to variables declared/used as Class Variables instead of variables are defined under Class Members.

Moving declared class variables in Class Members would help in resolving above ConcurrentModificationException in multi-threaded environment.


Wednesday, June 28, 2017

Performance Improvement To JSON Blobs Reads

The Dynamic Search feature is based on Amazon S3, RIAK KV as backend DB.
While performing Dynamic Search based with one or more than one field values, where large data objects/files are fetched from S3 buckets.

RIAK KV stores the metadata and link to S3 bucket for large data payload.
While searching for a data item with large json blob, ~100MB of data was sent to the client machine degrading the Search API performance.

The recommendation was to send meta data with link to the user and user can choose to download the item by clicking the link (each record of size 1MB, 2MB, 5MB or 10MB) when needed to see his observation data.

Wednesday, September 2, 2015

Deleting Duplicate Record In A Table

To delete duplicate records in a table use the below query:

delete /*+ PARALLEL(4) */ from letter_request A
WHERE a.rowid > 
   ANY (
     SELECT 
        B.rowid
     FROM 
        letter_request B
     WHERE 
      A.letter_name='WA20' and A.etl_batch_id=695
     AND
        A.loan_number = B.loan_number
     AND 
        A.sequence_number = B.sequence_number
        
     );

Wednesday, July 23, 2014

Capturing network packets using "tcpdump"

tcpdump is used for packet analyzer which allows the user to intercept and display TCP/IP and other packets being transmitted or received over a network to which the computer is attached.

Installation of tcpdump on Solaris/Linux:
   pkgutil is similar utility for Solaris compared to yum in Linux and apt-get in Ubuntu.
First let's install pkgutil using following command:
   /usr/sfw/bin/wget http://mirror.opencsw.org/opencsw/pkgutil.pkg
   pkgutil install at /opt/csw/bin/pkgutil on Solaris

Use below command to install tcpdump on Solaris:
pkgutil -i coreutils [This installs tcpdump at /usr/sbin/tcpdump]


Usage of tcpdump:
    tcpdump -w [output-file]  -i  [physical-interface] port [port]
    I noticed that capturing of the packets is only limited to 96 bytes. Checking manual of tcpdump I found that there is an option to disable the 96 bytes limitation by using the option "-s 0".  
Use the following command to capture 65535 bytes of packets.
    tcpdump -w [output-file] -i [physical-interface] port [port] -s 0

Example: tcpdump -w p001.pcap -i eth0 port 8080 -s 0
The above command captures any packet transfered on "physical-interface" on port 8080.

Analysis:
   The output file can be analyzed using "Wireshark" GUI application available for free.

Friday, April 18, 2014

Taking Thread Dumps At Regular Interval From CLI In Unix

Good to see posting some article on my blog after a long gap :)

Today I will explain on a very simple but important area needed for performance monitoring and analysis. This article is related to thread dump collection during performance execution in unix.

Topic: "How to collect thread dumps from CLI in Unix at regular interval"

Uuh... looks very simple post.. isn't it...!!! YES IT IS...

Use below command to capture thread dumps every 5 min:
nohup ./get_jstack  [Output-Log-Dir]  300  > /dev/null 2>&1 &

Now let's see what are the content of "get_jstack" file:

#!/bin/bash
LOGDIR=$1
WAIT=$2
mkdir $LOGDIR/serverjstack; mkdir $LOGDIR/muxjstack; mkdir $LOGDIR/wdjstack;
i=1
while [ $i -gt 0 ]
do
   PID1=`cat $LOGDIR/xmppd.pid`
   /usr/jdk/latest/bin/jstack $PID1 > $LOGDIR/serverjstack/jstack$i
   PID2=`cat $LOGDIR/mux.pid`
   /usr/jdk/latest/bin/jstack $PID2 > $LOGDIR/muxjstack/jstack$i
   PID3=`cat $LOGDIR/wd.pid`
   /usr/jdk/latest/bin/jstack $PID3 > $LOGDIR/wdjstack/jstack$i
    sleep $WAIT
    echo $i
    i=`expr $i + 1`
done

Make sure you provide the absolute path of jstack file. serverjstack, muxjstack and wdjstack are the folders for 3 processes which are running on the unix machine. 

here you go.. jstack will be created in the folder that you pass from command line. 
Enjoy Analyzing Thread Dumps For Performance Bottlenecks...

More to come soon.... :)

Thursday, July 12, 2012

Linux Multicore::Disable Core in Linux

We can disable the few cores from the multi-core linux machines by following the below commands:
$ sudo sh -c "echo 'n' > /sys/devices/system/cpu/cpu1/online"
E.g.
echo 0 > /sys/devices/system/cpu/cpu2/online; echo 0 > /sys/devices/system/cpu/cpu3/online;
The above line will disable the CPU 2 and 3 on a 4 CPU machine.
To enable it again:
 echo 1 > /sys/devices/system/cpu/cpu2/online; echo 1 > /sys/devices/system/cpu/cpu3/online;
The following command will display only enabled CPU:
cat /proc/cpuinfo

Wednesday, April 13, 2011

Calculating How Many Maximum Number Of Users Required If We Need To Achieve 12K Transactions With SLA Of 5 Minutes

Total Number of Transactions to be completed = 12,000
Average Response Time = 5 Minutes

To avoid putting much load on the system I will add 5 Minutes so that to total time taken to complete one iteration will be 10 Minutes. This addition will include the thinktime between the each screen/transaction traverse.

Now, I know each iteration will take 10 Minutes to complete, hence in 60 Minutes only 6 transactions can be completed. i.e. 1 User will do 6 transactions in an hour.

Now how to calculate how many users are required to complete 12,000 transactions in an hour?
The calculation will be:
#Users Required = 12,000/6
= 2000 Users

This way we can find the number of users required if total throughput is provided with an average response time.

Hence to achieve 12,000 transactions I need 2000 Users with 5 minutes of thinktime in between the iterations with an average response time of 5 Minutes.

Hope this post helps you to come-up with a Workload Modelling Concept (a bit). Please do let me know if my assumptions are wrong.

Calculating Pacing Time/Think Time to achieve 50 TPS with an average response time of 0.5 seconds with total of 100 Users

Let us start with calculating total number of transactions in an hour.
1 sec = 50 trnx
3600 sec = x
x = 3600 * 50 = 180000 trnx/hour by 100 Users

We have total number of users given as 100. Let us see how to calculate how many transactions each user will perform.
Total Number of Users = 100
Each User will perform 180000/100=1800 transactions/hour

Since, every transaction is taking on an average 0.5 seconds, let us see how much time is required to complete the each user transactions.
To complete 1800 trnx it will take 1800*0.5 = 15 minutes

So now, let us see how much think-time is required to complete the required number transactions per User in an hour.
1800 transactions will complete in 15 minutes
hence, 45 minutes of thinktime is required in between 1800 transactions (i.e. 45*60 = 2700 seconds of thinktime required in between 1800 transactions (per user))
2700 seconds = 1800 trnx
x = 1 trnx
x = 1.5 seconds think time need to include

Let us see how much time is required to complete each iteration.
Total time required to complete each Iteration = x + 0.5 seconds
= 1.5 + 0.5
= 2 seconds

Verification: Let us verify if our above calculation is correct.
Total time = 1800 * 2
= 3600 seconds
= 1 Hr

So, Each User will perform 1800 transactions where we will provide 2 seconds for each Iteration to complete.

Sunday, September 26, 2010

Bit On Performance Tuning

Let us have some information about tuning concept to be introduced during performance testing. In my scenario, the number of users were supported by the application was only 18. Beyond this number all users were failing to logging with the "Database Related Error". The error that I was getting was "Couldn't create Database Instance". [looks like it is a Database related error]
When I started looking into this issue, we had few areas that we could look for the solution. They were:
1. Connection Pools
2. Tomcat (if tomcat is the Web Server)
3. OS system file
4. Hardware resources

I encountered this issue on Solaris box. I started finding the solution for this issue and below are areas I looked into:

1. I started looking at the Application Server Connection pools issue. We had few parameters in the web.xml file under the middle-tier where min_connex and max_connex parameter was defined related to the Database. These parameters were set to 2 and 5 respectively. The best way to start finding the solutions for any issue is the log that is being written. In the middle-tier log, out of 4, 3 connection pools were always idle. So, there was no problem at the middle-tier side.

2. I tried increasing the tomcat memory from 512MB to 1024MB in catalina.sh file. This didn't resolve the issue.

3. Looked into the /etc/System file on the Solaris OS and saw some parameters like maxusers, maxuproc (maximum user processes). Setting maxusers to a value will control max_nprocs and maxuprc. The algorithms are:
max_nprocs = 10 + (16 x maxusers)
maxuprc = max_nprocs – reserved_procs (default is 5)
Even this setting didn't resolve the issue.

4. last but not the least, I looked into the System resources and found that the root path (i.e. "/") doesn't have much disk space to allocate DB resources any more beyond 18 users. Since the Oracle was setup at "/" path and the Oracle folder wasn't assigned any more system resources, Oracle was utilizing the space allocated to "/". Due to in-sufficient disk space on "/" path, users were not able to make connection with the Oracle DB and hence was failing.
Increasing the root disk space eliminated the issue with the user login.

Hope this was a good article to understand the tuning concept applied before looking into in details of performance tuning.

Friday, May 21, 2010

Difference In Monitoring System Resources From Performance Center & Controller

You might have monitored system resources from HP Performance Center or HP LoadRunner Controller application. But any time did you have faced any problem while collecting data (like %Processor Time, pages/sec etc for memory etc).
There is difference in adding measurements from Performance Center & from Controller. Have you ever seen that when you add System IP for collecting data from this IP while executing your scenario, but the data is not collected. I have faced this problem. Let me explain my scenario in this case:
In Performance Center:
1. You create a scenario, add scripts to execute in Design tabs and add the monitors in the Monitors tab (you provide the UserID/Password while adding the performance counters to be monitored on the machine).
2. You start the scenario, and the performance data is collected from this machine.

But do you really think it is so simple if you add performance counters for a machine on HP LoadRunner Controller.
Try adding the performance counter in LoadRunner Controller scenario and start the scenario. You will see that the window resources graph doesn't display the data. (it has happened with me) .
This is due to the fact that, while adding the machine under window resources, it doesn't ask for the system UserID/Password and the LoadRunner Controller machine UserID/Password overrides the UserID/Password of the system which is to be monitored.

So in this case, the password of the Controller machine and the Monitor machine should be same. Then only you can monitor the system resources added in the Controller scenario.

Saturday, April 24, 2010

Monitoring System Resources On Remote Machines Using PERFMON

We use to monitor the system resources using PERFMON and our machines without any issues, but I faced some different kind of issues while monitoring the remote machines using PERFMON tool from a centralized machine.

Before going into the detail of the PERFMON issues, please ensure few things on your machine where you are going to start the counter logs.
a) Check that Performance Logs and Alerts service is running
b) The service name is SysmonLog and the path to executable is "C:\window\system32\smlogsvc.exe". This service should be running in the services window.

I had 8 machines configured on the same domain and wanted to monitor these machine from a single point centralized machine. The required counters were created using "logman Create" method and to start and stop these perfmon counters I used to have a batch file. (This will monitor system resources on all the 8 machines from the centralized machine.

Now here comes the problem....!!!!

When I started the counter logs, it didn't start collecting data from these 8 machines. The reason for this is that, the user (for this centralized machine) doesn't have access to these machines so the perfmon counter logs were not getting started on these machines and hence the data is not collected.

The solution for this issue is that, after creating the counter logs on the centralized machine to monitor system resources of remote machines, perform the following actions:
1. Open Perfmon window
2. Go to properties of the counter logs (that has been created to monitor the remote machines)
3. On the System Overview properties (in XP, will be different in higher version of OS), type the User Name in the "Run As:" field and click on either "Set Password" or "Change" button (based on the OS version you are using)
4. On the Set Password window, provide the User Name, Password and Confirm Password and click on OK Button

Now, you are all set to monitor the system resources of the remote machine from a centralized machine.

Windows Excel Doesn't Support Big/Complex Formulas Spreadsheet To Open

You might have worked on MS Office Excel XP, 2003 or 2007. But did anyone tried opening an excel file of size more than 170 MB with lot of big and complex formulas in the excel cells?

If not, try once and then go below to read this article.

I faced this type of problem when I was working on one of my project and this project uses MS Excel to prepare the reports and these excel were of size almost more than 100 MB with lot of Excel complex formulas in it.

Excel doesn't work if there are lot of formulas in the spreadsheet. This is due to the fact that each excel cell formula occupies memory. And if you are trying to open these excel files with complex formulas the excel throws an error saying "Not enough resources".

Does someone knows why you get this error? If No, then here is the explanation......

This issue is due to the fact that this excel file has already reached the maximum memory limit per process on a 32-bit OS i.e 2GB per Process limit. This is the reason you will not be able to open an excel file that crosses 2GB memory on 32-bit OS.

This process can be eliminated if we use 64-bit OS with 64-bit Office Excel (if available), as 64-bit OS allows more than 2GB per process support.

In my case, I tried with Office 2010 on Windows 2008 or windows 7 and it worked (excel file has utilized more than 5 GB of memory)

So next time when you face this type of issue don't blame Office Excel but think broadly and try other alternatives.

Wednesday, March 3, 2010

Calculating Processor(_Total)\%Processor Time For System Using Process(*)\%Processor Time Performance Counter

Before going to detailed description on this topic, I would like to start it with a question...!!!!
"Can we calculate %Processor Time for the System which has multi-core processor if Processor(_Total)\%Processor Time counter is not included in the Perfmon?" [Also, provided we have all the necessary counters for the Process Object in the Perfmon, for e.g. Process(*)\%Processor Time is included]

Any Guess...!!!!!
Let me tell you how we can do this using the Process(*)\%Processor Time counters for the Process Object.
We can do this by using the following formula which will reflect exactly the same values for the Processor Counter [i.e. Processor(_Total)\%Processor Time]
Use the below formula:
Processor(_Total)\%Processor Time =[Process(_Total)\% Processor Time - Process(Idle)\% Processor Time]/Number Of Cores

Considering Process(_Total)\% Processor Time = Number_Of_Cores * 100

Bit confusing..????
Let me explain you with some example which will give you more understanding on this measurement.
Consider the below raw data which I have from the Perfmon utility every 1 second. I have considered here only 3 performance counters to make you understand:
1. Process(_Total)\% Processor Time
2. Process(Idle)\% Processor Time
3. Processor(_Total)\% Processor Time

Process(_Total)\ % Processor Time Process(Idle)\ % Processor Time
194.7602101 113.7708158
199.68128 174.72112
196.5612 170.04109
198.12127 174.72112

Processor(_Total)\% Processor Time
43.11458591
12.63944
14.97946
12.63944

Descriptions:
I have a Dual-Core machine, so let me consider the column \\MachineName\Process(_Total)\% Processor Time to evaluate it's values as:

\\MachineName\Process(_Total)\% Processor Time = 2 x 100 = 200 [So we can consider column values of this counter as 200 instead of 194.7602101, 199.68128 and so (this is nothing but the value 100% for each cores).

So, now let me calculate the Processor(_Total)\% Processor Time with the available Process related data.
e.g.
Processor(_Total)\% Processor Time = [200 - 113.7708158]/2
= 43.1145921 (which is almost equal to what it is shown in the third column of the Perfmon result i.e. equal to column values MachineName\Processor(_Total)\% Processor Time)

This explains the procedure to calculate the Processor(_Total)\% Processor Time if in case only Process counters are included in the Perfmon Utility.

Suggestions are always welcome........!!!!! :)

Monday, February 15, 2010

Copying Recent Updated Files From The Shared Drives To The Local Machine (From Same Network)

In case multiple log files are created on different physical machines connected on the same network, in my case the total physical machines were more than 30 and on each machine multiple log files were written by the application (say the average was 7-8 files). Even the size of the files does matter, as the size of the files written were ~700M and more.
Imagine copying these many files manually would always eat performance testers precious time. To eliminate this workload on the tester we can make use of the automated script to take care of copying the log files after each run.

What needs to be taken care:
1. If any services writes the log on these physical machine (same in my case), When the test is completed, execute the code which will collect the logs from different machines (my case: GAP service/PNL Service/DS/tradeserver logs).
2. The code should know how many logs and which logs it has to collect from which server.
3. The code sort the files based on the latest update (Date & Time) [latest files will be on top]
4. The code picks up the files which starts with text which is mentioned in the below If conditions.
5. The code picks up only the files which has the _error text in the name of the file
e.g. GAP1_BOND_FILT2_error_10142009_101408.log
HS1_BOND_FILT2_error_10142009_103028.log
6. The copied logs will be saved in the location that you provide during the execution of this code.

Code Goes Here:
package copyfiles_env2;
import java.io.*;
import java.io.File;
import javax.swing.*;
import java.util.Arrays;
import java.util.Comparator;

public class Main {
public static void main(String[] args) throws IOException
{
int forloop = 0;
boolean b1;
String string;
String[] servers={"\\\\server1b","\\\\server2b","\\\\server3b","\\\\server4b","\\\\server5b","\\\\server6b","\\\\server7b","\\\\server8b","\\\\server9b","\\\\server10b","\\\\server13b","\\\\server14b","\\\\server17b","\\\\server19b","\\\\server20b","\\\\server21b","\\\\server32b"};
BufferedReader TempReader;
BufferedWriter TempWriter;
File destFileName=null, DestFolderpath=null,tempfile=null;
String folderpathofscript = null, destfolderpath=null, temp1=null, s1=" ",s2=" "; // string to save the folder path selected by user

final JFileChooser fc = new JFileChooser();
fc.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY);
int returnVal1 = fc.showSaveDialog(null),fxpssearch2Count=0;
if (returnVal1 == JFileChooser.APPROVE_OPTION){
DestFolderpath = fc.getSelectedFile();
destfolderpath = String.valueOf(DestFolderpath);
JOptionPane.showMessageDialog(null, "Wait.....!!! Copying Files To The Destination Folder");

for (int j=0; j<=servers.length; j++){
folderpathofscript = servers[j]+"\\servers\\RTScripts";
if (servers[j]=="\\\\server1b"){
forloop = 2;
s1 = "GAP1_GAP_";
s2 = "GAP2_GAP_";
}
if (servers[j]=="\\\\server2b"){
forloop = 2;
s1 = "BVS_bondview_";
s2 = "BVS_collateralexceptionview_";
}
if (servers[j]=="\\\\server3b"){
forloop = 2;
s1 = "BVS_credderivdefaview_";
s2 = "BVS_dispexview_";
}
if (servers[j]=="\\\\server4b"){
forloop = 2;
s1 = "BVS_hedgeview_";
s2 = "BVS_incmsgview_";
}
if (servers[j]=="\\\\server5b"){
forloop = 2;
s1 = "BVS_requestsview_";
s2 = "BVS_riskview_";
}
if (servers[j]=="\\\\server6b")
{
forloop = 2;
s1 = "staticchangesserver_";
s2 = "tradeserver_";
}
if (servers[j]=="\\\\server7b"){
forloop = 2;
s1 = "GAP1_GAP_DERIV_";
s2 = "GAP2_GAP_DERIV_";
}
if (servers[j]=="\\\\server8b"){
forloop = 2;
s1 = "PRESTL_ACCT_TR_";
s2 = "PRESTL_BASKETOPT_";
}
if (servers[j]=="\\\\server9b"){
forloop = 2;
s1 = "PRESTL_EXOTIC_";
s2 = "PRESTL_FRA_";
}
if (servers[j]=="\\\\server10b"){
forloop = 2;
s1 = "PRESTL_MM_";
s2 = "PRESTL_MUST_TR_";
}
if (servers[j]=="\\\\server13b"){
forloop = 2;
s1 = "PNL1_PL_CDS_";
s2 = "PNL2_PL_CDS_";
}
if (servers[j]=="\\\\server14b"){
forloop = 2;
s1 = "PNL1_PL_BOND_";
s2 = "PNL2_PL_BOND_";
}
if (servers[j]=="\\\\server17b"){
forloop = 2;
s1 = "BDPS_";
s2 = "FXPS_";
}
if (servers[j]=="\\\\server19b"){
forloop = 2;
s1 = "HS1_BOND_";
s2 = "HS2_BOND_";
}
if (servers[j]=="\\\\server20b"){
forloop = 2;
s1 = "HS1_CDS_";
s2 = "HS2_CDS_";
}
if (servers[j]=="\\\\server21b"){
forloop = 2;
s1 = "HS1_DERIV_";
s2 = "HS2_DERIV_";
}
if (servers[j]=="\\\\server32b"){
forloop = 1;
s1 = "DS_";
}

File dir = new File(folderpathofscript);
File [] files = dir.listFiles();
Arrays.sort(files, new Comparator(){
public int compare(Object o1, Object o2) {
return compare( (File)o1, (File)o2);
}
private int compare( File f1, File f2){
long result = f2.lastModified() - f1.lastModified();
if( result > 0 ){
return 1;
} else if( result < 0 ){
return -1;
} else {
return 0;
}
}
});

for (int i=0;i<=forloop;i++){
if( files[i].getName().lastIndexOf(".log")!=-1 && files[i].getName().indexOf("error_")!=-1){
System.out.println("File Name: "+files[i].getName());
if(b1 = files[i].getName().indexOf(s1)!= -1 || files[i].getName().indexOf(s2)!= -1){
TempWriter = new BufferedWriter(new FileWriter(destfolderpath+"\\"+files[i].getName()));
TempReader = new BufferedReader(new FileReader(files[i].getPath()));
while((temp1=TempReader.readLine())!=null){
TempWriter.write(temp1+"\n");
}
TempWriter.close();
TempReader.close();
}
else if ((files[i].getName().lastIndexOf(".log")!=-1) && (files[i].getName().indexOf("STPAPI_")!=-1) || files[i].getName().lastIndexOf(".log")==-1 || files[i].getName().lastIndexOf("output_")==-1){
forloop = forloop + 1;
}
}
}
s1=" ";s2=" ";forloop=0;
}

JOptionPane.showMessageDialog(null, "Done.....!!! All The Required Files Are Copied Successfully....!!!");
}
}
}

Friday, January 8, 2010

Embedding Pacing In LoadRunner Script (Includes Pacing For Blocks Too)

We come accross many pacing logics in the LoadRunner script which is based on the multiple iterations. But if we need to have a pacing if there are Action blocks involved and if in case we need to re-calculate the pacing time after each iterations (which will be a good idea) as few iterations may not take the same time to execute in the LoadRunner Script, then, we need to implement a different Pacing Logic. Here it is.......
The below pacing logic calculates the pacing time after completion of each iterations and based on the test duration left.

This pacing logic have three different functions embedded in a .C function File.
1. InitCore()
2. TopOfIterations()
3. BottomOfIterations()

1. InitCore():
The InitCore() Function takes 3 parameters as mentioned in the below syntax:
InitCore(TestDuration(in Seconds), TotalBlockIterations, MaxNumberOfIterations);
Example: InitCore(180, 12, 2); - Script will execute for 24 iterations and for 180 seconds [i.e. 3 Min]
Refer the below screen for more understanding:

So, in this case the Total Number Of Iterations becomes 24 [TotalNoOfIterations = TotalBlockIterations + MaxNumberOfIterations].
This InitCore Function should be the first line in vuser_init section.

2. TopOfIterations():
This function calculates the Pacing Time based on the Test Duration and Total Number Of Iterations which will be used for the actual pacing time in the main script. This Function Call should be the first line in the Main Action Block.

3. BottomOfIterations():
This function does the main calculations of the Pacing Time when a Block Execution is completed based on the Test Durations left and the Total Number of Iterations left. This is same as re-calculating the Pacing Time instead of fixed Pacing Time each time the iterations is completed. Please note, here each BLOCK execution is considered as each ITERATIONS. This Function Call should be the last line in the Main Action Block.

These TopOfIterations() and BottomOfIterations() functions can be added in different Actions Blocks.

Bottleneck:
Since the Pacing Time is calculated based on the Test Durations left and Iterations left, there could be a possibility that few iterations might take long time to finish and if this is the case then at some point of time the Test Durations time becomes 0, in this case the rest of the Iterations are executed without calculating the Pacing Time and hence the Pacing Time becomes 0.

Actual Code Comes Here:

double TestDuration, pacing_time, Origpacing_time;
int Counter = 1,MaxCounter = 0, TotalIterations;
merc_timer_handle_t timer;
double duration;

InitCore(double TDuration, int TotalBlockIterations, int MaxNumberOfIterations)
{
TestDuration = TDuration;
TotalIterations =
TotalBlockIterations * MaxNumberOfIterations;
}

TopOfIterations()
{
if (Counter == 1) {
lr_output_message("Action1 - Total Test Durations - First Time: %lf",TestDuration);
pacing_time = (double)TestDuration / TotalIterations;
lr_output_message("Action1 - Total Iterations Left - First Time: %d",TotalIterations);
lr_output_message("Action1 - Pacing - First Time: %lf ",pacing_time);
}
}

BottomOfIterations()
{
if ((duration <> 0) {
TestDuration = TestDuration - pacing_time; //Is equal to Pacing time
MaxCounter = 1;
}

TotalIterations = TotalIterations - 1;
if (MaxCounter != 0) {
lr_think_time(pacing_time - duration);
pacing_time = TestDuration / TotalIterations;
MaxCounter = 0;
}
else if (TestDuration > 0) {
TestDuration = TestDuration - duration;
pacing_time = TestDuration / TotalIterations;
}

if (TestDuration <= 0) {
pacing_time = 0;
}
lr_output_message("Action1 - Total Iterations Left: %d",TotalIterations);
lr_output_message("Action1 - Total Test Durations Left: %lf",TestDuration);
lr_output_message("Action1 - New Pacing: %lf",pacing_time);
Counter = 0;
}
Enjoy Scripting.........