secondary_scrape_prompt.py

PROMPT = """You are an expert at extracting both explicit and implicit predictions about artificial intelligence from text. Your task is to perform a detailed second pass analysis to identify predictions that may have been missed in the first extraction pass.

The source year for these predictions is: {source_year}

Previously identified predictions from the first pass:
{existing_predictions}

OBJECTIVES:
1. Find ADDITIONAL concrete predictions about AI and its impacts that weren't captured in the first pass
2. Pay special attention to:
   - Implicit predictions embedded in descriptive statements
   - Conditional predictions ("if X happens, then Y will follow")
   - Comparative predictions ("X will happen before Y")
   - Negative predictions ("X won't happen until...")
   - Predictions about limitations or barriers
   - Timeline-based predictions
   - Economic and societal impact predictions

PREDICTION CRITERIA:
A valid prediction must:
1. Be related to artificial intelligence or its effects on society, economics, or technology
2. Be specific enough to be verifiably true or false when evaluated
3. Have an identifiable timeframe or resolution point
4. Be distinct from all previously extracted predictions
5. Represent a genuine forecast rather than a current state description

EXTRACTION METHODOLOGY:
1. First Pass: Identify statements that contain temporal indicators:
   - Explicit years or dates
   - Relative time references ("within X years", "by the time...")
   - Milestone-based timing ("once X happens", "after Y occurs")

2. Second Pass: Look for implicit predictions in:
   - Discussions of current limitations
   - Comparisons with previous technologies
   - Expert opinions or consensus views
   - Risk assessments
   - Investment or funding discussions
   - Technical capability forecasts

3. Third Pass: Analyze for predictions about:
   - Development timelines
   - Economic impacts
   - Social changes
   - Technical capabilities
   - Regulatory changes
   - Market adoption
   - Safety and risk scenarios

RESOLUTION YEAR DETERMINATION:
1. For explicit timeframes: Add the stated period to {source_year}
2. For implicit timeframes: Consider:
   - Technical dependencies mentioned
   - Economic or market factors
   - Regulatory requirements
   - Infrastructure needs
3. For uncertain timeframes: Use conservative estimates based on context

<important>
OUTPUT FORMAT:
Return a JSON object with a "predictions" array. Each prediction should be an object with:
<output>
{{
    "concrete_prediction": string,  // Clear, testable statement
    "resolution_year": number,      // Year for evaluation
}}
</output>
</important>

QUALITY CHECKS:
Before finalizing each prediction, verify:
1. It's not redundant with existing predictions
2. It's specific enough to be falsifiable
3. It has a clear resolution condition
4. It's properly categorized
5. The resolution year is justified by the context
6. The confidence level reflects the certainty of the prediction in the text

EXAMPLES OF VALID PREDICTIONS:
1. Explicit timeline: "AI systems will achieve human-level performance in medical diagnosis by 2030"
2. Implicit capability: "Current ML architectures won't be sufficient for full self-driving without orders of magnitude more data"
3. Economic impact: "Training costs for large language models will become economically unsustainable by 2028"
4. Social change: "AI assistants will be used in over 50% of office workflows by 2027"

EXAMPLES OF INVALID PREDICTIONS:
1. Too vague: "AI will continue to improve"
2. No timeline: "Eventually AI will be very powerful"
3. Current state: "AI systems are getting better at natural language"
4. Opinion: "AI development should be regulated"

Remember to:
- Prioritize precision over quantity
- Include supporting context for each prediction
- Assign confidence levels based on the strength of the evidence
- Categorize predictions consistently
- Verify each prediction is truly distinct from existing ones

Return your findings in valid JSON format."""