Skip to content

Latest commit

 

History

History
271 lines (257 loc) · 7.85 KB

91.md

File metadata and controls

271 lines (257 loc) · 7.85 KB
Raw

Results of GPT-4 on HumanEval/91

Date: 18 July, 2023
Model: gpt-4
Temperature: 0.2
Passing runs: 0
Total runs: 10
Success rate: 0%

Prompt

Tokens: 133

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """

The above is an incomplete Python code fragment. Return the complete and correct code with no additional text.

Correct solution

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """import re
    sentences = re.split(r'[.?!]\s*', S)
    return sum(sentence[0:2] == 'I ' for sentence in sentences)

Unit tests

def check(candidate):

    # Check some simple cases
    assert candidate("Hello world") == 0, "Test 1"
    assert candidate("Is the sky blue?") == 0, "Test 2"
    assert candidate("I love It !") == 1, "Test 3"
    assert candidate("bIt") == 0, "Test 4"
    assert candidate("I feel good today. I will be productive. will kill It") == 2, "Test 5"
    assert candidate("You and I are going for a walk") == 0, "Test 6"

    # Check some edge cases that are easy to work out by hand.
    assert True, "This prints if this assert fails 2 (also good for debugging!)"

Runs

❌ Run 1

Start time: 2023-07-18 18:19:18
Duration: 16.72s
Completion tokens: 219
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 2

Start time: 2023-07-18 18:19:01
Duration: 17.09s
Completion tokens: 204
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    for sentence in sentences:
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 3

Start time: 2023-07-18 18:18:44
Duration: 16.7s
Completion tokens: 198
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 4

Start time: 2023-07-18 18:18:35
Duration: 9.16s
Completion tokens: 138
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 5

Start time: 2023-07-18 18:18:17
Duration: 17.57s
Completion tokens: 194
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 6

Start time: 2023-07-18 18:17:51
Duration: 25.79s
Completion tokens: 261
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 7

Start time: 2023-07-18 18:17:38
Duration: 13.56s
Completion tokens: 138
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 8

Start time: 2023-07-18 18:17:20
Duration: 17.44s
Completion tokens: 223
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 9

Start time: 2023-07-18 18:17:09
Duration: 11.63s
Completion tokens: 138
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))

❌ Run 10

Start time: 2023-07-18 18:16:52
Duration: 16.21s
Completion tokens: 203
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    sentences = [s.strip() for s in re.split('[.!?]', S)]
    return sum(1 for s in sentences if s.startswith('I'))