LoadRunner: Creating manual web_custom_request for HTTP file uploads

There was a recent forum post in Advanced LoadRunner yahoo group asking how LoadRunner captures the file upload requests and if there was a way to customize that request. At that time, I replied that it may be possible to generate a manual web_custom_request by using the underlying raw HTTP request. I hadn't tried it then but it sounded interesting enough to be marked as to-do. I recently got some time to try it and it was fun.

Problem:

As mentioned in the forum post, LoadRunner captures any form based file uploads as web_submit_data with the file name (and location) within the list of data within the request. An example request:

web_submit_data("FileUpload",
        "Action={URL}",
        "Method=POST",
        "EncType=multipart/form-data",
        "TargetFrame=",
        "RecContentType=text/html",
        "Mode=HTML",
        ITEMDATA,
        "Name=File", "Value=C:\\testdata\\readme1.txt", "File=yes", ENDITEM,
        LAST);

There are 2 obvious drawbacks to this:

1. For this kind of script, the files to be uploaded have to be transferred to each of the Load Generator machines being used in the scenario. Or they have to be transferred to a shared network location so that both the machine used for creating the script and the load generator machines can reference it.

2. As mentioned in the forum post, any parameterization in the file contents is not possible. If the scenario requires files to be uploaded iteratively with unique names, that many files have to be created manually.

Solution:

There are 2 solutions to this and one is more universally acceptable (by web servers) than the other. I'll describe both and let you decide which one you want to use. I also highly recommend reading RFC1867 which provides excellent background on how multipart file uploads are implemented in HTML form based submissions.

1. web_custom_request POST: To arrive at the solution, I did exactly what I usually do when I'm stuck with some HTML script or want to see the raw HTTP request being sent to any web server. I opened Webscarab to capture the HTTP request that was being sent for file uploads. (Webscarab related post here).

Here's a screenshot of the HTTP post request as captured by Webscarab:

  

And here's the relevant portions of the raw request:

Content-Type: multipart/form-data; boundary=---------------------------327572003712859
Content-length: 228

-----------------------------327572003712859
Content-Disposition: form-data; name="File"; filename="readme1.txt"
Content-Type: text/plain

testdata
readme
readme
123456789
-----------------------------327572003712859--

The Content-Type HTTP header is what specifies the submission to be a multipart data submission. The "boundary" is a string that doesn't occur in the file data and is used to mark the boundaries of the data being sent. Content-length, as the name implies is the length of the data that is being sent.

The body of the request starts with the boundary string and is followed by the 2 headers to specify the content-disposition which includes the name of the file to be uploaded. Content-type specifies the encoding of the data being submitted. Following that within the body is a blank line and the actual contents of the file that will be uploaded. After the contents is the boundary once again to signify end of the submission data.

So this kinda makes it easier to create a web_custom_request. The Content-Type HTTP header is specified by "EncType" attribute of the function. Content-length header can be ignored since web_custom_request itself generates it by calculating the length of the data at runtime. The rest goes in the "Body" attribute starting with the boundary. Content-Disposition specifies among other things the name of the file to be uploaded and this can be parameterized. Content-Type is the type of file being uploaded and can be text/plain, image/gif etc. And in the end is the boundary once again. The final request looks like this:

web_custom_request("{rndString}.txt",
        "URL={testURL}",
        "Method=POST",
        "EncType=multipart/form-data; boundary=---------------------------17773322701763",
        "TargetFrame=",
        "Resource=1",
        "RecContentType=application/octet-stream",
        "Body=-----------------------------17773322701763\r\n"
        "Content-Disposition: form-data; name=\"File\"; filename=\"{rndString}.txt\"\r\n"
        "Content-Type: text/plain\r\n"
        "\r\n"
        "testdata\r\n"
        "readme\r\n"
        "readme\r\n"
        "{rndString}\r\n"
        "-----------------------------17773322701763--\r\n",
        LAST);       

I have parameterized the file name to be a random string so I can run this script iteratively without having to worry about server rejecting duplicate files. I can also parameterize the contents of the file within the body to whatever I like so that each file is unique or based on some other logic.

2. web_custom_request PUT: The second option is more straightforward but less universally accepted. I've read that not all web servers implement this. It involves using the HTTP PUT to create a web_custom_request. For more on different HTTP methods, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html. It mentions that:

The fundamental difference between the POST and PUT requests is reflected in the different meaning of the Request-URI. The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended and the server MUST NOT attempt to apply the request to some other resource.

I've also read that PUT is more efficient so if your test site implements the uploads using POST, better do it that way instead. Nevertheless, here is the web_custom_request with PUT:

web_custom_request("{rndString}.txt",
        "URL={URL}/{rndString}.txt",
        "Method=PUT",
        "Resource=1",
        "RecContentType=application/octet-stream",
        "Body=testdata\r\n"
        "readme\r\n"
        "readme\r\n"
        "{rndString}\r\n",
        LAST);

The resulting function is straightforward, URL attribute is the URI identifying the file to be uploaded and Body is the exact contents of the file.

Of Code Reviewers and Food Critics

For last 2-3 months, I was involved in testing our in-house customizations of Sun's IdM product. My responsibilities included not only leading the QA effort for the customizations but also to support the development effort by reviewing and suggesting improvements to the code/design.

So engrossed in my newfound glory as a unit tester/code reviewer, I realized that it may be easy for me to look at the code and find out a problem with it. But if I had to write something like that myself, I'd end up spending much more of my mind and time than it usually would take an average programmer. That realization reminded me of something I had recently heard in a movie.

The movie was "Ratatouille" and the scene was when Anton Ego starts writing his critique after visiting Gustaeu's Restaurant and having a meal cooked by the new but unknown chef Remy:

In many ways, the work of a critic is easy. We risk very little yet enjoy a position over those who offer up their work and their selves to our judgment. We thrive on negative criticism, which is fun to write and to read. But the bitter truth we critics must face, is that in the grand scheme of things, the average piece of junk is more meaningful than our criticism designating it so.

The parallels between a food critic as described above and what I was doing seemed surrealistically close, so much that I had to finish this post (after realizing it had been lying in my drafts for a few weeks). In some ways, that was exactly what I was doing. I enjoyed a position over those who wrote the code because finding out a problem with a piece of code drew much more attention than the original act of writing it. I thrived on negative criticism because it would get immediate credit, instead of the work of a developer which was the reason my role existed in the first place.

There are some fundamental differences as well. One being that Anton Ego, as a food critic does not have to care about the success of the Restaurant he is reviewing. This affords him the luxury of being overtly aggressive and have a dismissive attitude. My role on the other hand (both as a tester and code reviewer), is as responsible for the success of the product as any other stakeholder. That is the reason I'm as much pained when I find an issue in some code that I previously have reviewed and may have overlooked as the developer with whom I have to revisit the information once again. But the basic responsibility of both is still the same: to challenge the creator of an artifact to produce a better product which aligns with or exceeds the expectations of the users/consumers.

----------------------------------------------------------------------------------

The project is nearing its completion now and in hindsight, this is what I have to say:

This was a dream project for me. The kind of project that comes like once in a few years and offers amazing challenges with tremendous learning opportunities. Right kind of people, right kind of responsibilities and right kind of control over what I wanted to do. As much as I enjoyed working every moment in this project, I have to realize that in my enthusiasm, I may have stepped over some boundaries. I may have offered some unwarranted advices that were probably irrelevant or I may have criticized something to hide my lack of competence. I will take full responsibility for all the misgivings I caused, in however form I may deserve. But I do hope it will be realized that most of whatever I did was aimed at the common cause of project's success, and the fact that I enjoyed every bit of it and tried to make it enjoyable for everyone else should make it all the more worthwhile.

Calculating Virtual User Ramp up and Pacing for Load Tests

I was working on performance testing for a project not long ago that had different VUser types each with different load requirements. This meant that in one load test scenario, there were several user groups having unique transactional load characteristics (like desired TPM), not based on the whole scenario but by individual groups. At that time I created a simple spreadsheet to calculate the ramp-ups, pacing and actual load for each of the group and totals given the expected load and the number of virtual users. I recently got some time to improve it a little and am sharing it now.

So here's that table with some sample values:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	100	1	30	2.00	2	200	30.00
2	34.2	14	1	34.2	1.75	2	28	30.00
3	24.6	14	1	24.6	2.44	2	28	30.00
4	19.8	10	1	19.8	3.03	3	30	20.00
5	1.5	1	1	1.5	40.00	40	40	1.50
6	9.3	5	1	9.3	6.45	6	30	10.00
7	6.9	3	1	6.9	8.70	9	27	6.67
8	34.2	14	1	34.2	1.75	2	28	30.00
9	24.6	14	1	24.6	2.44	2	28	30.00
Total	185.1	175		185.1				188.17

A cleaner Google Spreadsheet read-only version is available here;
and an excel spreadsheet for your own use can be downloaded from here

How to use this table:

Basically, this table represents one load test scenario. So each of the row entry is a separate VUser group that is included in this scenario. Under VUser Group Column, the name of user group can be entered. If less or more VUser groups are needed to include in test scenario, rows can be removed or added accordingly.

There are 3 values that need to be entered for each user group: the base load that is desired for that user group, the number of virtual users that are to be used, and a multiplier (which in most cases can be 1). These are explained below:

Base Load: This is the load (in transactions per minute) at which that particular user group will be run. This is a user provided value based on actual requirements of the load test. The Total in last row gives the total load (in TPM) for this whole scenario. The total may or may not mean much but it does provide the total TPM at a glance.

Number of Virtual Users: This is the number of virtual users that will be used to run that user group. This is also a user provided value. Determining this will be a little trickier and there will have to be some estimation involved if you have a small number of VU licenses available (as I do). You will need the estimated time taken by a complete iteration of the transaction represented by this VUser group. Once you have that, you'll have to use enough virtual users to keep VU Pacing of this group above that number. Actually, you should use more Virtual users than that because in a load test, transaction time will vary depending on how stressed the test servers are and may be higher than the times you see when running a single user.

Let me use an example:
For my user group 1, I need to generate a load of 30TPM and I know by running the script with a single user that it takes around 10 seconds to complete one iteration. I want to keep the pacing much above 10 seconds, say at least 60 seconds. Its not difficult to see that I'll need to use 30 users to achieve a pacing of 60 seconds. I can just put 30 in the Number of VU Column and find out what the ramp up and pacing values will be.

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	30	1	30	2.00	2	60	30.00

Also, if more virtual users are available, they can be used based on the requirements and/or a desire to increase the VU pacing even more. So if I want to use 100 users for this user group, the numbers will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	100	1	30	2.00	2	200	30.00

On the other hand, if I have only 15 users available for this group, the values will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	15	1	30	2.00	2	30	30.00

Now to achieve 30TPM with 15 users, I will have to use a pacing of 30 seconds which is much closer to 10 seconds. But in real life with limited number of virtual users available, I may have to go with these values and hope that the iterations don't take more than 30 seconds during the test.

There is another thing to keep in mind: With this table, the values will be accurate if the expected load for each user group is less than 60. If it is more, the expected ramp up will be less than a second and it will be rounded off to 0 or 1. If it is zero, you'll get a division by zero error in Actual Load Column and if it is 1, you'll get the maximum Actual Load of 60.

So the thing to do if you want to calculate the values for loads above 60TPM is to break the group into 2 different groups. For example, to generate the values for 90TPM load, you can break it up into 2 groups of 60TPM and 30TPM. That way, you can generate 90TPM accurately. Other thing that can be done is to ramp up more than 1 user per interval. So with 90TPM, the expected ramp up will be 0.67 per user which means 3 users in 2 second ramp up scheme. This calculation will have to be done manually as the spreadsheet doesn't take care of fractional ramp up values and rounds them to nearest integer value. (see more of this under Actual Ramp Up)

Multiplier (Load Factor): This column solely exists because I had to run different tests at different loads (for example, a stress test that was run at 5 times the load of general load test) and I wanted to quickly find out how many users will be needed at different load factors and what will be their ramp up and pacing values. For example, if I had a user group with 1.5TPM load and 1 user with a pacing of 40 seconds and wanted to find out how many users will be needed to have the same pacing but with 5 times the load, I can use a Load Factor of 5 to find out that I'll need 5 users to maintain a pacing of 40 seconds, as shown in table below:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	1.5	5	5	7.5	8.00	8	40	7.50

So based on these 3 values entered for each user group, the remaining values, namely Ramp up time, Pacing and Actual TPM achieved are calculated.

Expected Load (TPM): This is calculated by simply multiplying base load with the load factor.

Expected Load (TPM): This is the theoretical ramp up time that is required to achieve the expected load and is calculated by dividing 60 by expected load. For example, if expected load is 30TPM, the ramp of course will be 60/30 = 2 users per second.

Actual Ramp up: Since the theoretical ramp up calculated in previous column can be a fractional value (indicating millisecond ramp up times) and we can have actual ramp up times only at a 1-second granularity, the actual ramp up is calculated by rounding off the expected ramp up to nearest integer. For example, if the expected load is 18TPM, the expected ramp up will be 3.33 seconds. This has to be rounded off to 3 seconds. This has the obvious effect of making the Actual Load different than the desired load.

Another case to watch out for, as I mentioned earlier is when the expected load is above 60TPM. In this case, the expected ramp up time is always going to be less than 1 and the actual ramp up will get rounded up to 0 or 1. The maximum Actual Load will be 60 as I'm not accounting for ramp ups greater than 1 user per second.

However, for both the above cases, there are 2 ways you can achieve the actual load that is closer to the desired load:

1. Break the group into multiple groups: For example, a 90TPM group can be broken into 2 groups - one with desired load of 60TPM and other with 30TPM. This way, an actual total TPM of 90 can be achieved.

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	60	60	1	60	1.00	1	60	60.00
2	30	30	1	30	2.00	2	60	30.00

In case of 18TPM, it can be broken into 2 groups of 15 and 3TPM. The values will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	15	30	1	15	4.00	4	120	15.00
2	3	6	1	3	20.00	20	120	3.00

2. More users per second ramp-up: Another way is to have multiple users per ramp up interval. For example, 90TPM with a single user group can be achieved by having a ramp up of 3 users every 2 seconds. Similarly, 18TPM can be achieved by having a ramp up of 3 users every 10 seconds.

I usually prefer the 1st method and choose the start time so that the ramp ups are even between the multiple groups of same user type. However it does make it more tedious as I have more user groups to keep account of. The 2nd method has the drawback of introducing a relative burst of users every ramp up interval.

VU Pacing: This is the interval between iterations and is calculated by multiplying the ramp up value with the number of users. Again, you have to make sure that the pacing value is comfortably above the single user transaction time; otherwise if the transactions take longer than the pacing value and next iteration is started immediately after the previous iteration, it may effect the load pattern.

Also, I'm assuming that you are pretty much content with having a stable pacing. If you want to add certain randomness to the the pacing times, that can also be done using this value. See my previous post for more on this here.

Actual Load: Finally, the Actual load is calculated by dividing 60 by the actual ramp up value. This is the actual load that you can hope to generate during your load test for this user group. What is actually generated during the load test depends on a lot of other factors like how stressed your system is etc. This can be found out by analyzing the results of the load test.

So that's that. It turned out to be a longer post than I expected, but hopefully it helps. If you have any comments or think of any suggestions to improve it, let me know.

Rubik Timer

I'm currently working on writing a Java based Rubik Timer. A Rubik timer is a specialized timer that can be used to track the time it takes to solve a Rubik's Cube. There are some implementations available currently but the ones that I tried lacked some much needed features. And since the current projects I'm working on are not proving challenging enough, I needed something else to keep my life exciting.

Even though I've written some code already and am faced with some design decisions, it came to my mind that it'll be a good opportunity to put myself in different shoes - as a requirements/business analyst and document the requirements, as a programmer/designer and design/develop the solution, as a tester and test the implemented solution and finally as a user and use the application. Inextricably entangled with all these roles will be a project management role that I will have throughout the project. So I plan to follow and document the project as it goes through its own lifecycle, in an attempt to learn more about different phases that a project goes through. In the process, I also want to understand the roles of all these different actors and the challenges they face. I know it is easier said than done and at times, it may seem like an undiscerning adventure. But I want to at least give it a try and even though the project is smaller in terms of effort required as compared to real life ones, I believe that it'll prove to be an enjoyable and learning experience. And the fact that I'll be playing all those roles will make it as challenging as (or even more than) those I face in real-life.

In upcoming related posts, I plan to detail the progress as I find time to work on this project. As it seems, resource availability will be the toughest challenge this project faces since my current projects, even though minimally challenging can be substantially time consuming and resource (in this case, my mind!) draining. And every now and then, smaller challenges have a tendency to pop-up and provide exciting distractions.