Setting up Custom Formatting
Custom Formatting is used when a log file is not recognized by Funnel Web Analyzer. Some logs are not recognized because of unusual Server configurations. In the following example, I will use an NCSA Extended format log file. This file is in something of a standard format for this type, but should serve to explain setting up a Custom Format. Below are four lines of the log separated by blank lines:
63.150.144.11 www.test.com - [23/Jul/2002:00:00:14 -0700] "GET /jobs/blue.html HTTP/1.1" 304 16 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)" "-"63.150.144.11 www.test.com - [23/Jul/2002:00:00:13 -0700] "GET /folio/flash/index.html HTTP/1.1" 304 84 "http://www.active.com/search/index.cfm" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)" "CFTOKEN=90067934; CFID=3228850; MYZIP=wa; BIGipServerevents-www=277911103.20480.0000; SERVERID=evapp1; ACTIVE_SESSION_TOKEN=evapp1102740759524500012878 BHYBWSUGKRPBAXKQSIOFDUDUTULQAWHP"
63.150.144.11 www.test2.com - [23/Jul/2002:00:00:13 -0700] "GET /images/headers/hdr_search.gif HTTP/1.1" 304 127 "http://www.active.com/search/index.cfm" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)" "CFTOKEN=90067934; CFID=3228850; MYZIP=wa; BIGipServerevents-www=277911103.20480.0000; SERVERID=evapp1; ACTIVE_SESSION_TOKEN=evapp1102740759524500012878 BHYBWSUGKRPBAXKQSIOFDUDUTULQAWHP"
63.150.144.11 www.test2.com - [23/Jul/2002:00:00:14 -0700] "GET /images/confirmation/invite_friends.gif HTTP/1.1" 304 36 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)" "-"
In the log line examples above the fields are separated by spaces. Sometimes it is difficult to tell if the separator is a space or a tab. A good way to do this is using an application such as TextPad. TextPad (Windows only) allows the user to View Visible Spaces in a Text file with a different symbol for space and tab. This application is available from : http://www.textpad.com/
To setup a Custom format for the log file, you first need to open the settings and select the Custom section. In the Custom format section, you need to select the checkbox to enable custom formatting. Firstly you need to specify the Field Delimiter and Date format. In the case of this log file, the field delimiter is space. To enter this, type space in the Field delimiter field. You can also type the ASCII equivalent of space, which is 32. The Date Format needs to be selected in the drop down also. In this log file, since the Time is included in the Date field, the Time format does not need to be selected. Finally, all of the fields in your log file need to be defined according to what position they occur in the log file. To do this:
On Windows: select the Log Field Type from the drop-down selector. Then drag the slider to the Field position that the Field occurs in.
On Macintosh: Select the Field Order position from the Drop down selector, then Select the associated Field from the Data drop-down selector. Then press the Add button to Add the field to the custom format. (To Delete fields, select in Log Fields window and select Delete button)
Identifying what the fields in log files represent can be difficult. It helps to have some knowledge of log files to interpret what the fields mean. Below I will interpret one of the log file lines above field by field to try and impart some knowledge in this area.
1. 63.150.144.11 : Numbers in this format represent the IP address of a Visitor to your site. In the Custom Format settings for this field, you would select Visitor in Position 1
2. www.test2.com : A web address in this format represents the home domain of the site hosted on the server. Where there are a number of sites hosted on the one server, different web addresses may appear in this field (as in the example above). In the Custom Format settings for this field, you would select Virtual Host in Position 2
3. - : Dashes such as this in the log file indicating there is no data recorded in this field. You do not need to enter any Custom settings for this field if no data found in log file
4. [23/Jul/2002:00:00:13 -0700] : This field displays the Date and time of the hit on the site in Greenwich Mean Time. The -700 value indicates that the location of the server is in a time zone seven hours behind Greenwich Mean Time in London. The space in this field will be ignored. Since this field contains the date and time, you do not need to specify the time field in the Custom settings separately. In the Custom Format settings for this field, you would select Date in Position 4.
5. "GET /folio/flash/index.html HTTP/1.1" : This field defines the page or file being accessed by the Visitor, as well as the Method and Protocol used to get the file or page. Since this field contains the Method (GET), you do not need to specify the Method in the Custom settings separately. The Protocol used (HTTP/1.1) also does not need to be defined in the Custom settings separately. The spaces in this field will be ignored. In the Custom Format settings for this field, you would select File in Position 4.
6. 304 : This field shows the Status of the transfer. eg. The number 404 here would indicate a broken link. In the Custom Format settings for this field, you would select Status in Position 5.
7. 84 : This field displays the number of bytes transferred. This field can usually be recognized by the variety of different whole numbers in this position. In the Custom Format settings for this field, you would select Bytes in Position 6.
8. "http://www.active.com/search/index.cfm" : This field displays the URL which Referred the visitor to the file or page. This field can usually be recognized since the entire URL is displayed. In the Custom Format settings for this field, you would select Referral in Position 8.
9. "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)" : This field displays the User-Agent and System Information of the Visitor to the site. Spaces in this field will be ignored and the System Info (Windows NT 5.0) does not need to be defined in the Custom settings. In the Custom Format settings for this field, you would select Agent in Position 9.
10. "CFTOKEN=90067934; CFID=3228850; MYZIP=wa; BIGipServerevents-www=277911103.20480.0000; SERVERID=evapp1; ACTIVE_SESSION_TOKEN=evapp1102740759524500012878 BHYBWSUGKRPBAXKQSIOFDUDUTULQAWHP" : This field displays a Cookie generated when accessing the site. This essentially defines various parameters for the site. You can choose to ignore this field or not. This field provides information on users. Should you choose to include it, you would select UserName in position 10. This cookie information will then appear in the Visitors report
Below is an example of how the Custom Settings for this log file would appear in the GUI for the Windows version:
Below is an example of how the Custom Settings for this log file would appear in the GUI for the Macintosh version:
When these settings are used, processing the log file should produce all valid reports
If you are using a command line version of Funnel Web Analyzer, such as Solaris or Linux, here is an example of how the settings file would appear when entered via the command line :
fwa45 -l /homedirectory/logfiles/customlogfile.log -usecustomformat -custom_seperator 32 -custom_dateformat 8 -custom_dataIndex 0 4 -custom_dataIndex 2 6 -custom_dataIndex 3 1 -custom_dataIndex 5 5 -custom_dataIndex 6 9 -custom_dataIndex 7 7 -custom_dataIndex 10 8 -custom_dataIndex 11 10 -custom_dataIndex 12 2The use of custom formatting via the command line versions involves writing complex commands but it does work. It is recommended that users create custom settings in a GUI version of the application and use these settings via the command line version using the -settingsfile or -r command
If you still want to use Custom settings through the command line, below is an explanation on how the commands used to drive the Custom format via the command line
-usecustomformat : This command turns on custom settings format for processing log files.
-custom_seperator X : Used to define the separator between fields used in the log file. Separator values (X) must be entered in ASCII format. See the table below for a translation of common separators into ASCII format:
Field Separator ASCII Evuivalent comma 44 tab 9 space 32
-custom_dateformat X : This field is used to define the Data Format (X)used in the log file. For an explanation of the numbers associated with each date format, see the table belowWhen processing log files via the command line versions of FunnelWeb Analyzer, using the commands above should utilise custom formatting correctly.
Date Format Numeric Value Fri 07 Mar 2002 1 DD/MMM/YYYY 2 MM/DD/YY 3 DD/MM/YY 4 YYYY-MM-DD 5 Thu Mar 1 10:32:18 2002 6 Unix Time 7 [06/May/2000:14:59:29 -400] 8 -custom_timeformat X : This field is used to define the Time Format (X)used in the log file. For an explanation of the numbers associated with each time format, see the table below
Time Format Numeric Value HH:MM:SS 1 1:40 PM 2 Unix Time 3 -custom_dataIndex X Z: This command is used to define the positions of all the different fields in the log file, where X = the Log field type ansd Z = the position in the log file. In the table below is a translation of the number equivalent of the Log Field Types to be entered in the X position:
Log Field Type Numeric Value Date 0 Time 1 Status 2 Visitor 3 Source 4 File 5 Agent 6 Bytes 7 BytesIn 8 Duration 9 Referral 10 UserName 11 VirtualHost 12 Method 13 Protocol 14 Port 15
Greg Stehle
Funnel Web Analyzer
This site looks best using Safari or FireFox.
