OSWbb – How to install and configure OSWatcher black box for system diagnostics

System troubleshooting is the major task for admin in day to day activities.They are facing difficulties to identify system performance issue on Linux.

There are many tools available in Linux to identify system performance issue but each tool (like top, vmstat, ps, iostat, etc) has it own features and we can’t get all in one place to identify the root causes.

Few tools are available in Linux to get all these details in one place. We already covered such kind of topic in past called sosreport.

Suggested Read :
(#) What Is sosreport? And How It Helps To Troubleshoot Issue In Linux
(#) Sysdig – Linux System Exploration/Diagnostic And Troubleshooting Tool

What is OSWatcher (OSWbb)

Oracle OSWatcher Black Box (OSWbb) collects and archives operating system and network metrics that you can use to diagnose performance issues. It’s a set of shell scripts OSWbb operates as a set of background processes on the server and gathers data on a regular basis, invoking below Unix utilities.

Make sure the blow packages should be installed on your system as prerequisites. If not, install missing packages using distribution package manager.

  • VMSTAT
  • IOSTAT
  • MPSTAT
  • IFCONFIG
  • NETSTAT
  • TOP
  • TRACEROUTE

oswbb is comprised of two separate components:

  • oswbb: An Unix shell script data collector which collects and stores the data.
  • oswbba: A java utility which will analyze the data automatically and provide advice and produce graphs and an html document.

How to install OSWatcher (OSWbb) in Linux

This is straight forward. Just download the latest available package version from oracle website and extract to descried location.

Move/Copy the file to the directory where you want to install OSWbb and extract the file.

# tar xvf oswbb812.tar

It creates a directory called oswbb, which contains all the directories and files that are associated with OSWbb.

# ls -lh /opt/oswbb
total 516K
drwx------  2 oracle oinstall 4.0K Sep  6  2017 analysis
drwx------ 13 oracle oinstall 4.0K Mar  1 18:03 archive
drwx------ 13 oracle oinstall 4.0K Oct 20 13:49 archive-old
-rwxr-xr-x  1 oracle oinstall   67 Jan 15  2014 call_du.sh
-rwxr-xr-x  1 oracle oinstall   68 Oct  7  2013 call_sar.sh
-rwxr-xr-x  1 oracle oinstall   71 Jan  7  2014 call_uptime.sh
drwx------  2 oracle oinstall 4.0K May  2  2017 data
drwx------  4 oracle oinstall 4.0K Sep  5  2017 docs
-rwxr-xr-x  1 oracle oinstall  665 Feb  6  2017 Example_extras.txt
-rwxr-xr-x  1 oracle oinstall 1.9K Oct  7  2013 Exampleprivate.net
-rwxr-xr-x  1 oracle oinstall 3.9K Apr 17  2017 genprvnet.sh
drwx------  2 oracle oinstall 4.0K Sep  6  2017 gif
-rwxr-xr-x  1 oracle oinstall  772 May  8  2014 ifconfigsub.sh
-rwxr-xr-x  1 oracle oinstall  743 Oct  7  2013 iosub.sh
drwx------  2 root   root     4.0K Mar 16 19:59 locks
-rwxr-xr-x  1 oracle oinstall 1.5K Jan  8  2014 ltop.sh
-rwxr-xr-x  1 oracle oinstall  542 Oct  7  2013 mpsub.sh
-rwxr-xr-x  1 oracle oinstall  740 Oct  7  2013 nfssub.sh
-rwxr-xr-x  1 oracle oinstall 6.7K Jul 12  2017 OSWatcherFM.sh
-rwxr-xr-x  1 oracle oinstall  47K Aug 14  2017 OSWatcher.sh
-rw-r--r--  1 oracle oinstall  10K Sep  6  2017 oswbb810.tar
-rwxr-xr-x  1 oracle oinstall 310K Sep  6  2017 oswbba.jar
-rwxr-xr-x  1 oracle oinstall  414 Oct  7  2013 oswib.sh
-rwxr-xr-x  1 oracle oinstall  435 Jan  8  2014 oswnet.sh
-rwxr-xr-x  1 oracle oinstall  825 Oct  7  2013 oswrds.sh
-rwxr-xr-x  1 oracle oinstall  524 Oct  7  2013 oswsub.sh
-rwxr-xr-x  1 oracle oinstall 1.5K Aug 24  2017 psmemsub.sh
-rwxr-xr-x  1 oracle oinstall  557 Jul 12  2017 sarsub.sh
drwx------  7 oracle oinstall 4.0K Sep  6  2017 src
-rwxr-xr-x  1 oracle oinstall 2.6K Feb 26  2015 startOSWbb.sh
-rwxr-xr-x  1 oracle oinstall  751 Aug 14  2017 stopOSWbb.sh
-rwxr-xr-x  1 oracle oinstall  834 Feb  6  2017 tar_up_full_archive.sh
-rwxr-xr-x  1 oracle oinstall 5.3K Feb  6  2017 tar_up_partial_archive.sh
drwx------  2 root   root     4.0K Mar 16 19:59 tmp
-rwxr-xr-x  1 oracle oinstall  527 Feb  7  2017 topaix.sh
-rwxr-xr-x  1 oracle oinstall  545 Feb 23  2015 vmsub.sh
-rwxr-xr-x  1 oracle oinstall 1.5K Feb 26  2015 xtop.sh

To start OSWbb, run the startOSWbb.sh script from the oswbb directory. This script accepts two parameters, which is Frequency & Duration.

  • Frequency : How frequent (in seconds) data needs to be collected.
  • Duration : How many hours data needs to be archived

If you do not input any arguments, the script runs with default values of 30 and 48. It collects data every 30 seconds and store the last 48 hours of data in archive files.

# ./startOSWbb.sh [Frequency Duration]

The below example collects and stores the last 10 hours of data to archive files at 60 second intervals.

# ./startOSWbb.sh 60 10
Setting the archive log directory to/opt/oswbb/archive

Testing for discovery of OS Utilities...
VMSTAT found on your system.
IOSTAT found on your system.
MPSTAT found on your system.
IFCONFIG found on your system.
NETSTAT found on your system.
TOP found on your system.
TRACEROUTE found on your system.
 
Discovery of CPU CORE COUNT
CPU CORE COUNT will be used by oswbba to automatically look for cpu problems

CPU CORE COUNT = 6
VCPUS/THREADS = 24

Discovery completed.

Starting OSWatcher v8.1.0  on Tue Feb 27 15:14:54 MST 2018
With SnapshotInterval = 60
With ArchiveInterval = 10

OSWatcher - Written by Carl Davis, Center of Expertise,
Oracle Corporation
For questions on install/usage please go to MOS (Note:301137.1)
If you need further assistance or have comments or enhancement
requests you can email me [email protected]

Data is stored in directory: /opt/oswbb/archive

Starting Data Collection...

oswbb heartbeat:Tue Feb 27 15:14:59 MST 2018
oswbb heartbeat:Tue Feb 27 15:15:59 MST 2018
oswbb heartbeat:Tue Feb 27 15:17:00 MST 2018
oswbb heartbeat:Tue Feb 27 15:18:00 MST 2018
oswbb heartbeat:Tue Feb 27 15:19:00 MST 2018
oswbb heartbeat:Tue Feb 27 15:20:00 MST 2018
oswbb heartbeat:Tue Feb 27 15:21:00 MST 2018
oswbb heartbeat:Tue Feb 27 15:22:00 MST 2018
.
.

To stop OSWbb prematurely, run the stopOSWbb.sh script from the oswbb directory.

# ./stopOSWbb.sh

How to view the collected data

Collected data will be stored under archive directory, it is created when OSWbb is started for the first time. OSWbb stores data in hourly archive files named system_name_utility_name_timestamp.dat, and each entry in a file is preceded by the characters *** and a timestamp.

# ls -lh /opt/oswbb/archive
total 44K
drwx------ 2 root root 4.0K Mar  1 18:06 oswcpuinfo
drwx------ 2 root root 4.0K Mar 16 19:01 oswifconfig
drwx------ 2 root root 4.0K Mar 16 19:01 oswiostat
drwx------ 2 root root 4.0K Mar 16 19:01 oswmeminfo
drwx------ 2 root root 4.0K Mar 16 19:01 oswmpstat
drwx------ 2 root root 4.0K Mar 16 19:01 oswnetstat
drwx------ 2 root root 4.0K Mar  1 18:03 oswprvtnet
drwx------ 2 root root 4.0K Mar 16 19:01 oswps
drwx------ 2 root root 4.0K Mar 16 19:01 oswslabinfo
drwx------ 2 root root 4.0K Mar 16 19:01 oswtop
drwx------ 2 root root 4.0K Mar 16 19:01 oswvmstat

To check any data, just navigate to corresponding directory and view the corresponding file. For testing purpose, we printed vmstat file output.

# more oracle.2daygeek.com_vmstat_18.02.28.2300.dat
Linux OSWbb v8.1.0 oracle.2daygeek.com
SNAP_INTERVAL 60
CPU_CORES 6
VCPUS 24
OSWBB_ARCHIVE_DEST /opt/oswbb/archive
zzz ***Wed Feb 28 23:00:09 MST 2018
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
3 23 18027636 873140  11044 477948  101  138  5013  1964   93   13 14  2 67 17  0
2 17 18052168 878900  11044 481844 7636 24704 88672 58333 28214 41906  6  4 59 31  0
2 16 18065140 873880  11580 486796 5692 13868 71832 81249 27917 42840  8  3 61 29  0
zzz ***Wed Feb 28 23:01:09 MST 2018
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
3 21 18481600 868868  10896 507336  102  138  5037  1966   96   19 14  2 67 17  0
5 12 18495272 880956  10884 502060 3472 14188 296964 76683 25768 44284  6  3 61 30  0
2 13 18495680 874532  11000 504336 4620 1532 215072 43823 22749 40949  6  2 64 29  0
zzz ***Wed Feb 28 23:02:09 MST 2018
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
7 21 18840580 866384  10756 479816  102  139  5054  1967    0   25 14  2 67 17  0
5 10 18845460 870052  10868 483252 5624 5696 191656 75508 24249 42834  7  3 60 31  0
2  9 18855152 881676  10844 478320 5620 10556 248988 22601 21635 37511  5  2 66 28  0
zzz ***Wed Feb 28 23:03:10 MST 2018
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
8 12 15170720 1074652  18680 489808  103  139  5068  1967    3   31 14  2 67 17  0
2 11 15169920 1055024  19356 500372 5868    0 185516 202669 24888 45625  5  2 67 27  0
2 16 15169420 1037896  19640 509260 3944    0 99680 268204 25285 45869  4  2 67 27  0
zzz ***Wed Feb 28 23:04:10 MST 2018
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
8  8 15347816 879160   7268 595280  103  139  5058  1973    7   36 14  2 67 17  0
5  5 15346984 865644   7536 602356 6164    0 51172 52905 24542 39722 14  1 67 18  0
--More--(8%)

OSWatcher Analyzer

oswbba is a graphing and analysis utility which comes bundled with OSWatcher. This allows the user to graph and analyze data collected from running OSWatcher. Analyzing this data automatically avoids the time consuming task of manually inspecting all the files that OSWatcher collects. This utility is a stand-alone java program which requires java to be installed.

# java -jar oswbba.jar -i /opt/oswbb/archive/

Starting OSW Analyzer V8.1.2
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c)  2017 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...

Parsing file oracle.2daygeek.com_iostat_18.04.19.0300.dat ...
Parsing file oracle.2daygeek.com_iostat_18.04.19.0400.dat ...
This directory already exists. Rewriting...

Parsing file oracle.2daygeek.com_vmstat_18.04.19.0300.dat ...
Parsing file oracle.2daygeek.com_vmstat_18.04.19.0400.dat ...


Parsing file oracle.2daygeek.com_netstat_18.04.19.0300.dat ...
Parsing file oracle.2daygeek.com_netstat_18.04.19.0400.dat ...

Parsing file oracle.2daygeek.com_top_18.04.19.0300.dat ...
Parsing file oracle.2daygeek.com_top_18.04.19.0400.dat ...

Parsing file oracle.2daygeek.com_ps_18.04.19.0300.dat ...
Parsing file oracle.2daygeek.com_ps_18.04.19.0400.dat ...


Parsing Completed.


Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs

Enter GC to Generate All CPU Gif Files
Enter GM to Generate All Memory Gif Files
Enter GD to Generate All Disk Gif Files
Enter GN to Generate All Network Gif Files

Enter L to Specify Alternate Location of Gif Directory
Enter Z to Zoom Graph Time Scale (Does not change analysis dataset)
Enter B to Returns to Baseline Graph Time Scale (Does not change analysis dataset)
Enter R to Remove Currently Displayed Graphs

Enter X to Export Parsed Data to Flat File
Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

Please Select an Option:

About Vinoth Kumar

Vinoth Kumar has 3.5+ years of experience in Linux server administration & RHEL certified professional. He is currently working as a Senior L2 Linux Server administrator.

View all posts by Vinoth Kumar

Leave a Reply

Your email address will not be published. Required fields are marked *