Store Drupal logs on Amazon S3

ByKarthik Kumar D Kon24th Jun 2019, 2022-12-01T08:00:00+05:30
Read
Pause
Resume
Stop
Store Drupal logs on Amazon S3

Store Drupal logs on Amazon S3 via hook_watchdog, so that you can get rid of heavy logs on your drupal database and can later read from the S3.

For this todo on Drupal

  • You should use the "hook_watchdog" hook, where this hook allows modules to route log events to custom destinations.
  • In our case, our custom destination will be S3.
  • Initially we need to get the access to AWS and appropriate S3 bucket.
  • Store the connection configuration details of S3 buckets in the Drupal.
  • In the hook_watchdog, read the S3 Auth configs.
  • Create a connection to AWS S3, on success it return back with the S3 object.
  • Check if any buckets exist to write the logs, of not you should create bucket on S3 for logs.
  • Next, Create a log directory and log file, populate it with the data came on hook_watchdog.
  • Next, Write to the S3 bucket with parameters which "putObject" expects.
  • Next, Remove the log directory & file which is created in the process.
  • Check the S3 bucket via S3 UI, you could see the logs

Here's the piece of code, via we log to S3.

watchdog(    'module_2',    t('Info: Successfully completed.'),    array($data),    WATCHDOG_INFO);watchdog(    'module_3',    t('Error: Issue while processing.'),    array($exception, $data),    WATCHDOG_ERROR);

Here, the piece of code which helps to connect & write log's to S3.

<?phpuse \Aws\S3\S3Client;function audit_watchdog(array $log_entry) {    // Set the modules which needs to be logged to S3    $modules = array(        'module_1',        'module_2',        'module_3',    );    if (in_array($log_entry['type'], $modules)) {        if(isset($log_entry) && !empty($log_entry)) {            // Get the S3 config details            $s3_config = variable_get("s3_config");            $s3_bucket = $s3_config->s3_bucket;            $s3_region = $s3_config->s3_region;            $s3_key = $s3_config->s3_key;            $s3_secret = $s3_config->s3_secret;            try {                if (!empty($s3_bucket) && !empty($s3_region) && !empty($s3_key) && !empty($s3_secret)) {                    // Create AWS connection                    $s3 = create_aws_connection($s3_region, $s3_key, $s3_secret);                    if (!empty($s3) && is_object($s3) && $s3 != FALSE) {                        if ($s3->doesBucketExist($s3_bucket)) {                            $log_path = 'public://logs/to_aws';                            // Create log directory                            $dir_path = create_aws_log_directory($log_path);                            // Create log file                            $file_path = create_aws_log_file($log_entry, $dir_path);                            // Get the name of log file                            $keyname = get_keyname_for_log_file($log_entry['severity'], $file_path);                            // Store the log file to S3                            $s3->putObject([                                'Bucket' => $s3_bucket,                                'Key' => $keyname,                                'Body' => '',                                'SourceFile' => $file_path,                            ]);                            // Remove the directory & file created                            if (is_dir($dir_path)) {                                rmdir_recursive($dir_path);                                rmdir_recursive('public://logs');                            }                        }                    }                }            }            catch (Exception $e) {                // Exception is ignored so that watchdog does not break pages during the                // installation process or is not able to create the watchdog table during                // installation.            }        }    }}function create_aws_connection($s3_region, $s3_key, $s3_secret) {    $s3 = new S3Client([        'version' => 'latest',        'region'  => $s3_region,        'credentials' => [            'key'    => $s3_key,            'secret' => $s3_secret,        ]    ]);    $buckets = $s3->listBuckets()->get('Buckets');    if (isset($buckets) && !empty($buckets)) {        return $s3;    }    else {        return FALSE;    }}function create_aws_log_directory($log_path) {    if (!is_dir($log_path)) {        mkdir($log_path, 0777, true);        chmod($log_path, 0777);    }    return $log_path;}function create_aws_log_file($log_entry, $dir_path) {    $content = json_encode($log_entry);    $log_file_name = $log_entry['type'] . '-' . date("Y-m-d-H-i-s") . '-' . preg_replace("/^.*\./i","", microtime(true)) . '.log';    $file_path = $dir_path . '/' . $log_file_name;    $log_file = fopen($file_path, "w");    $write_log_file = fwrite($log_file, $content);    $close_log_file = fclose($log_file);    $chmod_log_file = chmod($file_path, 0777);    return $file_path;}function get_keyname_for_log_file($severity, $file_path) {    $watchdog_array = array(        "0" => "WATCHDOG_ERROR",        "1" => "WATCHDOG_INFO",    );    return 'drupal-logs/' . $watchdog_array[$severity] . '/' . basename($file_path);}function rmdir_recursive($dir) {    foreach(scandir($dir) as $file) {        if ('.' === $file || '..' === $file) continue;        if (is_dir("$dir/$file")) rmdir_recursive("$dir/$file");        else unlink("$dir/$file");    }    rmdir($dir);}

Advantage of having Logs on S3

  • Reduce the number of DB log entries on Drupal database.
  • Completely Keep the Audit system outside Drupal, So Prod instance will play smooth.

Cheers :)

26,991OFF
Hisense 139 cm (55 inches) Bezelless Series 4K Ultra HD Smart LED Google TV 55A6H (Black)

Hisense 139 cm (55 inches) Bezelless Series 4K Ultra HD Smart LED Google TV 55A6H (Black)

32,999 59,990
Resolution : 4K Ultra HD (3840x2160) | Refresh Rate : 60 Hertz | 178 Degree wide viewing angle. Connectivity: 3 HDMI ports to connect set top box, Blu Ray players, gaming console | 2 USB ports to connect hard drives and other USB devices | Dual-band Wi-Fi | Bluetooth 5.1. Sound : 24 Watts Output | Dolby Atmos | Dolby Digital. Smart TV features: Google TV | Watchlist | Google Assistant | Far Field Vioce Control | Chromecast, Miracast, DLNA | Auto Low Latency Mode for VRR | Supported Apps : Netflix, Youtube, Prime Video, Hotstar, SonyLiv, Hungama, JioCinema, Zee5, Eros Now. Display :10 bit Panel | Bezel-less Floating Display Design | ALLM | Decoding of Dolby Vision, HDR10, HLG | 1 Billion Colours | MEMC.
Televisions
350OFF
Amazon Brand - Solimo Slim Stainless Steel Water Bottle, Set of 3, 1 L Each

Amazon Brand - Solimo Slim Stainless Steel Water Bottle, Set of 3, 1 L Each

649 999
Stainless Steel water bottle set for home and office use. Made using high quality stainless steel for added durability and long life. Made from 100% food grade materials to be safe for everyday usage. Features a spill-proof design; has a snug fitting lid for easy usage and prevention of leaks. Ergonomic design for comfortable grip and rust free body. Ideal for storing water, iced tea, juices, shakes and more. Dimensions: L x W x H: 7.3 x 7.3 x 26 cm. Package Contents:- Solimo Stainless Steel Water Bottle Set of 3, Weight: 200 gm each, Colour: Silver, Capacity: 1 litre each.
Kitchen & Dining
1,272OFF
Bajaj Rex 500W Mixer Grinder, Purple

Bajaj Rex 500W Mixer Grinder, Purple

2,278 3,550
Wattage : 500 W, Voltage: 230V~50 Hz: Revolution: 20,000 RPM. Powerful 500W Titan Motor with Radio Knob: Easy selection of Speed. 2-in-1 Blade: The dry grinding jar can be used for dry and chutney grinding both. Product warranty by Bajaj - 2 Years. T&C Applied. Pulse Mode: Used for mixing and grinding without stopping your mixie. Perfect to mince chicken , get bread crumps, small puree and making lump free gravies.
Kitchen Appliances

Related Articles

32,991OFF
Hisense 139 cm (55 inches) Tornado 2.0 Series 4K Ultra HD Smart LED Google TV 55A7H (Silver)

Hisense 139 cm (55 inches) Tornado 2.0 Series 4K Ultra HD Smart LED Google TV 55A7H (Silver)

36,999 69,990
Resolution : 4K Ultra HD (3840x2160) | Refresh Rate : 60 Hertz | 178 Degree wide viewing angle. Connectivity: 3 HDMI ports to connect set top box, Blu Ray players, gaming console (HDMI 1 eARC supported) | 2 USB ports to connect hard drives and other USB devices | Dual-band Wi-Fi | Bluetooth 5.1. Sound : 102 Watts Output | JBL 6 Speakers System | Dolby Atmos for remarkable sound quality. Smart TV features: Google TV with Watchlist | Google Play Store | Google Assistant | Chromecast, Miracast, DLNA, Airplay | Auto Low Latency Mode for VRR | Supported Apps : Netflix, Youtube, Prime Video, Hotstar, SonyLiv, Hungama, JioCinema, Zee5, Eros Now. Display :10 bit Panel | Bezel-less Floating Display Design | ALLM | Decoding of Dolby Vision, HDR10, HLG | 1 Billion Colours | MEMC. Warranty Information: 2 Year Comprehensive Warranty on product provided by Hisense from date of purchase.. Installation: For requesting installation/wall mounting/demo of this product once delivered, please directly call Hisense Support for assistance (Please visit Hisense Website for Toll Free Numbers) and provide product's model name and seller's details mentioned on your invoice. The service center will allot you a convenient slot for the service. Easy Returns: This product is eligible for replacement within 10 days of delivery in case of any product defects, damage or features not matching the description.
Televisions

Recent Articles

Recent Quick Read

Recent Great People

1,059OFF
Butterfly Abs Hippo Tabletop Wet Grinder 2 Litre, 150 W, 230 V, Ac 50Hz, Cherry Red

Butterfly Abs Hippo Tabletop Wet Grinder 2 Litre, 150 W, 230 V, Ac 50Hz, Cherry Red

3,998 5,057
Wattage: 150 W; Voltage: 220-240V, 50-60Hz ; Revolution: 1300. No of Jars: 1; Jar Size: 2 Liters; Jar Material: Stainless Steel; Body Material: ABS body; Cord Material: 180 CM; Speed Control: 1 Speed Control. Read Manual Before Use. Powerful motor; Product Dimensions: 47.5 x 30.5 x 28 cm; Uniform grinding pressure and 4 Way grinding system. Shock proof ABS body, Feature: Detachable Stainless Steel drum & roller stones. Sturdy single phase high torque motor; Unbreakable and transparent lids made of polycarbonate material; Includes: Main Unit, 2 stones, lid, Warranty card with user manual.
Kitchen Appliances
802OFF
Bajaj RCX 1.8 DLX Rice Cooker, 1.8 Litre , White

Bajaj RCX 1.8 DLX Rice Cooker, 1.8 Litre , White

1,898 2,700
1.8-Litre capacity; 1Kg raw rice cooking capacity. The Cooking Bowl is Aluminum with anodized finish. Automatic cooking. Keep Warm : No. Warranty: 2 years on product. Power: 700 watts; Operating voltage: 230 volts.
Kitchen Appliances
4,171OFF
Amazon Brand - Solimo Xxxl Bean Bag Filled With Beans (Black And Brown)(Faux Leather)

Amazon Brand - Solimo Xxxl Bean Bag Filled With Beans (Black And Brown)(Faux Leather)

1,829 6,000
Dimensions: 71D x 71W x 124H Centimeters; Weight: 3600 Grams. Beans Required: No. Pattern: Faux Leather| Style: With Beans. Special Features: Velcro and zipper offer double protection to prevent beans from spilling out. Warranty: 1 year on manufacturing defects | Specific use: Bedroom, Lounge. Assembly Instruction: No assembly required.
Furniture
601OFF
amazon basics Vacuum Compression Storage Bags with Hand Pump - Medium, 5-Pack, Plastic

amazon basics Vacuum Compression Storage Bags with Hand Pump - Medium, 5-Pack, Plastic

599 1,200
Set of heavy-duty plastic vacuum compression storage bags. Compress bulky textiles, clothing, linens, and more at an 80% size reduction. Protects items from moisture, dirt, insects, mold, odors, and more. Hand pump included, or connect to any standard vacuum cleaner hose for faster results. Heavy-duty valve and double seal zipper for air tight protection. 5-pack of Medium bags.
Household
We Need Your Consent
By clicking “Accept Cookies”, you agree to the storing of cookies on your device to enhance your site navigation experience.
I Accept Cookies