Evaluation System – Approach Doc

Step 1: Define Objectives and Requirements

Objectives:

1. Evaluate website performance, SEO compliance, and technical integrity.
2. Identify and document issues.
3. Provide actionable steps to repair identified issues.
4. Track the status of repairs and evaluations.

Requirements:

1. Integration with existing tools (e.g., Screaming Frog, Google PageSpeed Insights).
2. User-friendly interface for documentation and tracking.
3. Automated reporting and notifications.
4. Compatibility with various CMS platforms (e.g., WordPress, Plesk).

Step 2: Choose Tools and Technologies

Tools and Technologies:

1. Airtable or Trello: For tracking and managing tasks, issues, and status.
2. Screaming Frog: For SEO audits, broken link checks, and indexing verification.
3. Google PageSpeed Insights: For performance evaluation.
4. W3C Validator: For HTML validation.
5. SSL Checker (e.g., Why No Padlock): For SSL and mixed content checks.
6. Custom Scripts: For specific checks and automation (e.g., checking redirects, canonical tags).
7. Plesk/WordPress Admin Panels: For direct management and repairs.
8. Zapier or Integromat: For automating workflows between tools.

Step 3: Develop the Process

Process Overview:

1. Initial Evaluation:
   • Run Screaming Frog for an SEO audit and to identify broken links.
   • Use Google PageSpeed Insights for performance checks.
   • Validate HTML using W3C Validator.
   • Check SSL configuration with SSL Checker.
   • Review technical configurations on Plesk or WordPress.
2. Documentation and Task Management:
   • Document issues in Airtable or Trello.
   • Create tasks for each identified issue with detailed steps for resolution.
   • Assign tasks to relevant team members.
3. Repair and Validation:
   • Implement fixes based on the documented issues.
   • Re-run evaluations (Screaming Frog, PageSpeed Insights, W3C Validator) to ensure issues are resolved.
   • Update the status of tasks in Airtable or Trello.
4. Final Review:
   • Conduct a comprehensive review to ensure all issues are resolved.
   • Prepare a final report summarizing the evaluation and repair process.

Step 4: Detailed Workflow

1. Set Up Tracking System:
   • Create a base in Airtable or a board in Trello with columns for task status (To Do, In Progress, Done).
   • Create fields for task details (Section, Item Evaluated, Observations, Repairs Needed, Status).
2. Run Initial Checks:
   • Screaming Frog: Export SEO audit report.
   • Google PageSpeed Insights: Record mobile and desktop scores.
   • W3C Validator: List HTML validation errors.
   • SSL Checker: Document SSL certificate status and mixed content issues.
3. Document and Assign Tasks:
   • Enter each identified issue into Airtable or Trello.
   • Assign each task to a team member with a due date.
   • Provide detailed instructions for fixing each issue.
4. Repair Issues:
   • Team members work on assigned tasks, implementing fixes.
   • Update the task status as work progresses.
5. Re-Evaluation:
   • After repairs, re-run the initial checks.
   • Ensure all issues are resolved and document the results.
6. Final Review and Reporting:
   • Conduct a final review of the website.
   • Compile a report summarizing the findings, repairs, and final status.
   • Share the report with stakeholders.

Step 5: Automate and Optimize

Automation:

• Use Zapier or Integromat to automate task creation in Airtable or Trello based on outputs from tools like Screaming Frog and Google PageSpeed Insights.
• Set up automated notifications and reminders for task deadlines and status updates.

Optimization:

• Continuously refine the process based on feedback.
• Update the TNT Knowledge Base with new findings and best practices.
• Train new team members using documented processes and tools.

Airtable Template for Data Collection

Screaming Frog API Integration

Automate the identification and resolution of issues found by Screaming Frog, by integrating various tools and developing custom scripts to streamline the process.

Step 1: Set Up Screaming Frog for Automation

1. Configure Screaming Frog:
   • Set up a custom configuration in Screaming Frog to focus on the issues you want to identify (e.g., SEO issues, broken links, redirects, canonical tags).
   • Save the configuration for reuse in automated runs.
2. Enable Scheduling:
   • Use the scheduling feature in Screaming Frog to run crawls at regular intervals (e.g., daily, weekly).
   • Save the crawl reports in a designated folder or integrate with a cloud storage service like Google Drive or Dropbox.

Step 2: Extract and Process Data

1. Extract Data:
   • After each crawl, Screaming Frog can automatically export the crawl report in various formats (CSV, Excel).
2. Process Data:
   • Use a script (e.g., Python) to process the exported data. The script should:
     • Parse the exported report.
     • Identify and categorize issues (e.g., broken links, missing meta descriptions).
     • Generate a summary of findings.

Step 3: Integrate with Airtable or Trello

1. Create Tasks Automatically:
   • Use an automation tool like Zapier or Integromat to watch the folder where Screaming Frog exports reports.
   • When a new report is detected, trigger the script to process the report and create tasks in Airtable or Trello.
   • Include details such as the URL, issue type, and specific repair actions.
2. Task Management:
   • Ensure tasks are assigned to the appropriate team members with deadlines.
   • Set up notifications and reminders for task updates.

Step 4: Automate Common Repairs

1. Broken Links:
   • Use a script to automatically identify broken links and suggest potential replacements or fixes.
   • For internal links, the script can update the CMS (e.g., WordPress) database directly if permissions allow.
2. Redirects:
   • Generate .htaccess rules or server-side redirects based on identified issues.
   • Use a script to update the .htaccess file or server configuration automatically.
3. Missing or Duplicate Meta Descriptions:
   • Use a script to update missing or duplicate meta descriptions based on page content.
   • Integrate with the CMS API to apply changes directly.

Step 5: Re-Evaluation and Continuous Monitoring

1. Re-run Screaming Frog:
   • Schedule a re-crawl after automated repairs are made.
   • Verify that issues are resolved.
2. Continuous Monitoring:
   • Set up continuous monitoring and regular re-crawls.
   • Ensure any new issues are detected and addressed promptly.

Step 6: Reporting and Optimization

1. Generate Reports:
   • Use the processed data to generate reports summarizing the status and resolution of issues.
   • Share reports with stakeholders regularly.
2. Optimize the Process:
   • Gather feedback from team members to improve the automation scripts and workflows.
   • Update the TNT Knowledge Base with any new findings or optimized procedures.

Python Script for Processing Screaming Frog Reporting

pythonCopy codeimport pandas as pd
import os
from airtable import Airtable

# Configuration
screaming_frog_export_folder = 'path/to/export/folder'
airtable_base_key = 'your_airtable_base_key'
airtable_table_name = 'Tasks'
airtable_api_key = 'your_airtable_api_key'

# Initialize Airtable
airtable = Airtable(airtable_base_key, airtable_table_name, airtable_api_key)

# Function to process Screaming Frog report
def process_screaming_frog_report(file_path):
    df = pd.read_csv(file_path)
    issues = []

    # Example: Identify broken links
    broken_links = df[df['Status Code'] == 404]
    for index, row in broken_links.iterrows():
        issues.append({
            'URL': row['Address'],
            'Issue': 'Broken Link',
            'Details': f"Link to {row['Address']} returns 404",
            'Status': 'To Do'
        })

    # Add other issue identification logic here...

    return issues

# Function to create Airtable tasks
def create_airtable_tasks(issues):
    for issue in issues:
        airtable.insert(issue)

# Main function to process reports and create tasks
def main():
    for file_name in os.listdir(screaming_frog_export_folder):
        if file_name.endswith('.csv'):
            file_path = os.path.join(screaming_frog_export_folder, file_name)
            issues = process_screaming_frog_report(file_path)
            create_airtable_tasks(issues)
            print(f"Processed and created tasks for report: {file_name}")

if __name__ == "__main__":
    main()

Integrate with Zapier or Integromat

1. Trigger:
   • New file in folder (Google Drive/Dropbox).
2. Action:
   • Run the Python script using a serverless function platform (e.g., AWS Lambda, Google Cloud Functions).
   • Create tasks in Airtable or Trello based on the processed data.