Store Drupal logs on Amazon S3

ByKarthik Kumar D Kon24th Jun 2019, 2022-12-01T08:00:00+05:30

Store Drupal logs on Amazon S3 via hook_watchdog, so that you can get rid of heavy logs on your drupal database and can later read from the S3.

For this todo on Drupal

You should use the "hook_watchdog" hook, where this hook allows modules to route log events to custom destinations.
In our case, our custom destination will be S3.
Initially we need to get the access to AWS and appropriate S3 bucket.
Store the connection configuration details of S3 buckets in the Drupal.
In the hook_watchdog, read the S3 Auth configs.
Create a connection to AWS S3, on success it return back with the S3 object.
Check if any buckets exist to write the logs, of not you should create bucket on S3 for logs.
Next, Create a log directory and log file, populate it with the data came on hook_watchdog.
Next, Write to the S3 bucket with parameters which "putObject" expects.
Next, Remove the log directory & file which is created in the process.
Check the S3 bucket via S3 UI, you could see the logs

Here's the piece of code, via we log to S3.

watchdog(    'module_2',    t('Info: Successfully completed.'),    array($data),    WATCHDOG_INFO);watchdog(    'module_3',    t('Error: Issue while processing.'),    array($exception, $data),    WATCHDOG_ERROR);

Here, the piece of code which helps to connect & write log's to S3.

<?phpuse \Aws\S3\S3Client;function audit_watchdog(array $log_entry) {    // Set the modules which needs to be logged to S3    $modules = array(        'module_1',        'module_2',        'module_3',    );    if (in_array($log_entry['type'], $modules)) {        if(isset($log_entry) && !empty($log_entry)) {            // Get the S3 config details            $s3_config = variable_get("s3_config");            $s3_bucket = $s3_config->s3_bucket;            $s3_region = $s3_config->s3_region;            $s3_key = $s3_config->s3_key;            $s3_secret = $s3_config->s3_secret;            try {                if (!empty($s3_bucket) && !empty($s3_region) && !empty($s3_key) && !empty($s3_secret)) {                    // Create AWS connection                    $s3 = create_aws_connection($s3_region, $s3_key, $s3_secret);                    if (!empty($s3) && is_object($s3) && $s3 != FALSE) {                        if ($s3->doesBucketExist($s3_bucket)) {                            $log_path = 'public://logs/to_aws';                            // Create log directory                            $dir_path = create_aws_log_directory($log_path);                            // Create log file                            $file_path = create_aws_log_file($log_entry, $dir_path);                            // Get the name of log file                            $keyname = get_keyname_for_log_file($log_entry['severity'], $file_path);                            // Store the log file to S3                            $s3->putObject([                                'Bucket' => $s3_bucket,                                'Key' => $keyname,                                'Body' => '',                                'SourceFile' => $file_path,                            ]);                            // Remove the directory & file created                            if (is_dir($dir_path)) {                                rmdir_recursive($dir_path);                                rmdir_recursive('public://logs');                            }                        }                    }                }            }            catch (Exception $e) {                // Exception is ignored so that watchdog does not break pages during the                // installation process or is not able to create the watchdog table during                // installation.            }        }    }}function create_aws_connection($s3_region, $s3_key, $s3_secret) {    $s3 = new S3Client([        'version' => 'latest',        'region'  => $s3_region,        'credentials' => [            'key'    => $s3_key,            'secret' => $s3_secret,        ]    ]);    $buckets = $s3->listBuckets()->get('Buckets');    if (isset($buckets) && !empty($buckets)) {        return $s3;    }    else {        return FALSE;    }}function create_aws_log_directory($log_path) {    if (!is_dir($log_path)) {        mkdir($log_path, 0777, true);        chmod($log_path, 0777);    }    return $log_path;}function create_aws_log_file($log_entry, $dir_path) {    $content = json_encode($log_entry);    $log_file_name = $log_entry['type'] . '-' . date("Y-m-d-H-i-s") . '-' . preg_replace("/^.*\./i","", microtime(true)) . '.log';    $file_path = $dir_path . '/' . $log_file_name;    $log_file = fopen($file_path, "w");    $write_log_file = fwrite($log_file, $content);    $close_log_file = fclose($log_file);    $chmod_log_file = chmod($file_path, 0777);    return $file_path;}function get_keyname_for_log_file($severity, $file_path) {    $watchdog_array = array(        "0" => "WATCHDOG_ERROR",        "1" => "WATCHDOG_INFO",    );    return 'drupal-logs/' . $watchdog_array[$severity] . '/' . basename($file_path);}function rmdir_recursive($dir) {    foreach(scandir($dir) as $file) {        if ('.' === $file || '..' === $file) continue;        if (is_dir("$dir/$file")) rmdir_recursive("$dir/$file");        else unlink("$dir/$file");    }    rmdir($dir);}